SERA integration on Fastly¶

Kindly follow the below steps to setup the routing of bot traffic to SERA.

Prerequisites¶

All your page requests must get intercepted by below SERA routing rule
- Please check your routing, caching and cache-variation policies for that
If caching is enabled, Users and Bots should have a separate cache-variations configured. This is to not serve the SERA's bot-optimized-content to users.

Steps¶

Kindly follow the below examples to setup the routing of bot traffic to SERA.

Change SERA_ENDPOINT to the SERA endpoint shared with you
Change YOUR_TOKEN to the SERA token shared with you
Keep the relevant bot user-agent regex condition mentioned below - to control the traffic being sent to SERA

Create a backend in the Fastly service¶

Clone the currently active version and create a backend for SERA.
You can use the following example code snippet for creating a SERA backend in your Fastly service.

backend o_s {
    .always_use_host_header = true;
    .between_bytes_timeout = 5s;
    .connect_timeout = 10s;
    .dynamic = true;
    .first_byte_timeout = 45s;
    .host = "SERA_ENDPOINT";
    .host_header = "SERA_ENDPOINT";
    .port = "443";
    .ssl = true;
    .ssl_check_cert = never;
    .ssl_sni_hostname = "SERA_ENDPOINT";
    .probe = {
        .dummy = true;
        .initial = 5;
        .request = '"HEAD / HTTP/1.1"  "Host: SERA_ENDPOINT" "Connection: close";'
        .threshold = 1;
        .timeout = 2s;
        .window = 5;
    }
}

Route Crawler requests to SERA¶

Use the same draft intercept and route the crawler requests to SERA.
Please refer to the following VCL snippet to intercept and route the required crawler requests to SERA in vcl_recv subroutine.
You need to add the following snippet in addition to your previous configuration(if any).
Note: Do not replace vcl_recv subroutine as it is if you have any existing configuration.

sub vcl_recv {
    # only send PAGE requests to SERA
    if (req.method == "GET" 
        && req.url.ext ~ "^(?i)(|html|php|aspx)$"
        && !querystring.get(req.url, "nsbp")
        && req.url !~ "(?i)(/static/|/_next/|/api/|/login|/logout|/account/|/wishlist/|/cart/|/checkout/|/payment/).*"
        && !req.http.x-nv-app
        && !req.http.x-nv-sera-bypass
        && req.http.User-Agent ~ "MERGE AND USE THE BELOW USER_AGENT REGEXES AS PER REQUIREMENT") {
        # DURING POC PHASE - (?i)(n7TestUserAgent|google\ page\ speed|chrome-lighthouse|google-inspectiontool|PTST|GTmetrix|n7ua)
        # ON GOING LIVE - (?i)(n7TestUserAgent|google\ page\ speed|chrome-lighthouse|google-inspectiontool|PTST|GTmetrix|n7ua|googlebot|facebookexternalhit|bingbot)
        # ON GOING LIVE for AI bots - (?i)(openai|gptbot|claudebot|claude-user|perplexity-user|perplexity-bot|google-extended|google-cloudvertexbot|google-notebooklm|amazonbot|duckassistbot|metaai|metabotic|meta-externalagent|copilot|baiduspider)

        set req.url = querystring.remove(req.url); /* REMOVE THIS IF YOUR SITEMAP PAGE URLS HAVE QUERYSTRING */
        set req.http.X-NV-Sera-Token = {YOUR_TOKEN};
        set req.http.X-NV-Access-Key = {YOUR_N7_ACCESS_KEY};
        set req.backend = F_o_s;

        return(lookup);
    }

    # existing configuration
}

You can test the Fastly configuration on Fiddle
Keep refining the logic by adjusting user-agent list, and SERA exclusion-patterns as needed.
Test thoroughly:
- WPT and GTMetrix will automatically receive the response from SERA due to the routing rule set above.
  - Steps to validate page using WPT
- To check the SERA rendered page on browser, you can use DevTools > Network conditions facility, or some browser extension (like Simple modify headers) to manipulate the browser user-agent.
  If in testing phase, use these user-agent values:
  For desktop:
```
n7TestUserAgent
```
  For mobile:
```
n7TestUserAgent; Android
```
  If SERA is live, you can use these actual bot user-agent values:
  For desktop:
```
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +<http://www.google.com/bot.html)> Chrome/W.X.Y.Z Safari/537.36
```
  For mobile:
```
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +<http://www.google.com/bot.html)>
```
On go-live day, set the user-agent condition regex to actual bot user-agent values. Additionally, ensure that your condition is case-insensitive.
Whitelist SERA requests at your origin to avoid any issues with indexing. Details are here
After go-live, in any dashboards created for monitoring the "actual user traffic" e.g. Google analytics, exclude the requests with user-agent string containing word "Nitrogen SERA" to get correct understanding about user-traffic.

SERA integration on Fastly¶

Prerequisites¶

Steps¶

Create a backend in the Fastly service¶

Route Crawler requests to SERA¶

Testing and Refinement¶