Prerendering Overview

What is Prerendering?

Prerendering or "flattening" is a process that creates a static HTML version of your site's pages which bots may more readily ingest. Once your website is translated, GlobalLink Web crawls and caches the flattened HTML of the pages and makes them available through an API.

Prerendering is done only for in-scope URLs, and this scope is defined by choosing a sitemap. Your GlobalLink Web Implementation Team will be able to help with taking your sitemap and adding it for the prerendering service.

Prerendering Checklist

• Select URLs which should be flattened and create a sitemap (include all the language variations)
• Upload these urls to the Flattening UI and queue them up for rendering
• Setup and test bot sorting rules.
• Once the URLs are rendered, upload a multilingual sitemap to google search console and trigger a crawl
• Check google search console

How To Use Prerendering for SEO Optimization?

For bots to see the flattened version, traffic must be filtered by user-agent; bots are silently routed to the API location which presents the flattened page. This traffic sorting occurs at the CDN (content delivery network) or WebServer (e.g., NGNIX) level.

API Documentation

The Flatten API is specified by GET www.onelink-edge.com/xapis/Flatten/P{project_key}?url={url-encoded-originalurl}[&debug=1]

Based on the rendered/cached status and the original http status of the URL, the following responses may be seen:

Implementing BotSorting

To implement bot sorting, you can use your own CDN or server router to configure silent rewrite rules. Below are two different example methods: ProxyPass with Apache and CloudFront with AWS Lambda.

ProxyPass With Apache

The ProxyPass information needs to be updated when using the prerendering service so that search engine bots can find the translated versions of your pages. This update forces search engines to see the simplified translated versions of your site. Below is an example of the ProxyPass added to an Apache configuration. The Apache configuration below intercepts all user agents crawling the site for search engines and forwards them to our servers, which serve up simplified versions of all languages.

RewriteEngine      on

SSLProxyEngine     on

RewriteCond         %{HTTP_USER_AGENT} (.*[Bb]ot.*|.*[Ss]pider.*|.*slurp.*|.*archiver.*)

Example:                                                                                                                                                                                             (.*)  "https://www.onelink-edge.com/xapis/Flatten/P<key>?url=//%{HTTP_HOST}%{REQUEST_URI}"[P]

CloudFront with AWS Lambda

The second method utilizes an AWS Lambda serverless system, which can trigger certain events when the appropriate conditions are met. Lambda works alongside a content delivery network called CloudFront, which allows Lambda functions to be executed at edge locations. Below is a diagram of CloudFront events that can trigger a Lambda function.

When a viewer request is sent to the CloudFront cache for a website that utilizes GlobalLink Web, a Lambda function triggers to check whether the requesting user agent is a bot or a human user.

If the requesting user-agent is a bot, another Lambda function will trigger in the origin request to flatten the page and serve a simplified HTML page for the bot to index. The “rich” JavaScript page will be served if the user-agent is a human user.

One other Lambda function during the viewer request can trigger for sub-folder deployments only. If the entire site is not in scope, then the Lambda function checks to see if the page that is being requested is in scope or not. If the page is in scope, the second Lambda function will trigger during the origin request and flatten the page. If the page is not in scope, the second Lambda request will not trigger, and the page will remain rich.

To integrate this functionality, you will need to utilize Terraform, which allows you to create and update your Amazon web services infrastructure. Terraform will enable you to put the Lambda functions into the viewer and origin requests. The code snippets are included below.

Terraform Config Declaring Lambda Resource Ex.

Viewer Request Function:

'use strict';

/** Viewer Request **/

/*
 * This header MUST be included in a CloudFront cache policy to create separate cache entries for bots and
 * regular users and both this header and the Host header MUST be included in an origin request policy.
 * Both of these policies must be enabled on the CloudFront distribution as well.
 *
 * How to create a CloudFront cache policy:
 * https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/controlling-the-cache-key.html
 * How to create a CloudFront origin request policy:
 * https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/controlling-origin-requests.html
 */
const customHeaderName = 'OneLink-JS-Prerender';

exports.handler = (event, context, callback) => {
   const { Records: [{ cf: { request } }] } = event;
   const { headers, uri } = request;
   const { 'user-agent': [{ value: userAgent } = {}] = []} = headers;

   const userAgentRegex = new RegExp(/([Bb]ot|[Ss]pider|slurp|archiver)/);

   /*
     * This regex is for sub-folder deployments only. Make sure to include all lang/locale pairs.
     * Also, this regex is redundant in cases where CloudFront distributions are correctly configured 
     * with pattern matching behaviors: 
     * https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html?icmpid=docs_cf_help_panel#DownloadDistValuesPathPattern
     */
   const uriRegex       = new RegExp('^/({language folders bar seperated})');

   headers[customHeaderName.toLowerCase()] = [{
      key: customHeaderName,
      value: (uriRegex.test(uri) && userAgentRegex.test(userAgent)).toString()
   }];

   callback(null, request);

Origin Request Function:

'use strict';

/** Origin Request **/
const projectKey = '____-____-____-____';
const flattenKey = `P${projectKey}`;
// const TransPerfectDomain = 'www.onelink-edge.com'; // <- The current EC2 XAPIS
// const TransPerfectDomain = 'xapis.onelinkjs.com';  // <- The container XAPIS

exports.handler = (event, context, callback) => {
   const { Records: [{ cf: { request } }] } = event;
   const { headers, uri } = request;
   const {
      host: [{ value: host }],
      'onelink-js-prerender': [{ value: oneLinkJSPrerender }]
   } = headers;

   if (oneLinkJSPrerender === 'true') {
      request.querystring = `url=//${host}${uri}`;

      /* Set custom origin fields */
      request.origin = {
         custom: {
            domainName: TransPerfectDomain,
            port: 443,
            protocol: 'https',
            path: '',
            sslProtocols: ['TLSv1', 'TLSv1.1'],
            readTimeout: 5,
            keepaliveTimeout: 5,
            customHeaders: {}
         }
      };

      request.uri = `/xapis/Flatten/${flattenKey}`;
      headers.host = [{ key: 'host', value: TransPerfectDomain }];
   }

   callback(null, request);
};