Prerendering Overview

What is Prerendering?

Prerendering is a process that supplies bots with a stripped-down HTML version of your site that’s easy for them to crawl and users with a fully optimized JavaScript version that is more attractive and usable. Once your website is crawled (indexed) for translation with GlobalLink Web, a flattened version of your site is available through an API set. This version is a plain HTML representation of your original page’s content that has been translated into the target language.

How To Use Prerendering for SEO Optimization?

The general concept of prerendering is to serve up a simplified HTML page when the requesting user-agent is a known SEO bot or SEO bot IP. If the requesting user-agent is a user, the website serves up a “rich” JavaScript page. Suppose the website has a simplified GlobalLink Web page, but JavaScript is detected. In that case, it will redirect to the rich experience to prevent real users from accidentally seeing the simplified version.

This traffic sorting occurs at the CDN (content delivery network) or WebServer (e.g., NGNIX) level. The table below is an example of this process.

Prerendering API Documentation

GET www.onelink-edge.com/xapis/Flatten/P{project_key}?url={url-encoded-originalurl}[&debug=1]

Implementing Prerendering

To implement prerendering, you can use two different methods: ProxyPass with Apache and CloudFront with AWS Lambda.

ProxyPass With Apache

The ProxyPass information needs to be updated when using the prerendering service so that search engine bots can find the translated versions of your pages. This update forces search engines to see the simplified translated versions of your site. Below is an example of the ProxyPass added to an Apache configuration. The Apache configuration below intercepts all user agents crawling the site for search engines and forwards them to our servers, which serve up simplified versions of all languages.

RewriteEngine      on

SSLProxyEngine     on

RewriteCond         %{HTTP_USER_AGENT} (.*[Bb]ot.*|.*[Ss]pider.*|.*slurp.*|.*archiver.*)

Example:                                                                                                                                                                                             (.*)  "https://www.onelink-edge.com/xapis/Flatten/P<key>?url=//%{HTTP_HOST}%{REQUEST_URI}"[P]

CloudFront with AWS Lambda

The second method utilizes an AWS Lambda serverless system, which can trigger certain events when the appropriate conditions are met. Lambda works alongside a content delivery network called CloudFront, which allows Lambda functions to be executed at edge locations. Below is a diagram of CloudFront events that can trigger a Lambda function.

When a viewer request is sent to the CloudFront cache for a website that utilizes GlobalLink Web, a Lambda function triggers to check whether the requesting user agent is a bot or a human user.

If the requesting user-agent is a bot, another Lambda function will trigger in the origin request to flatten the page and serve a simplified HTML page for the bot to index. The “rich” JavaScript page will be served if the user-agent is a human user.

One other Lambda function during the viewer request can trigger for sub-folder deployments only. If the entire site is not in scope, then the Lambda function checks to see if the page that is being requested is in scope or not. If the page is in scope, the second Lambda function will trigger during the origin request and flatten the page. If the page is not in scope, the second Lambda request will not trigger, and the page will remain rich.

To integrate this functionality, you will need to utilize Terraform, which allows you to create and update your Amazon web services infrastructure. Terraform will enable you to put the Lambda functions into the viewer and origin requests. The code snippets are included below.

Terraform Config Declaring Lambda Resource Ex.

Viewer Request Function:

'use strict';

/** Viewer Request **/

/*
 * This header MUST be included in a CloudFront cache policy to create separate cache entries for bots and
 * regular users and both this header and the Host header MUST be included in an origin request policy.
 * Both of these policies must be enabled on the CloudFront distribution as well.
 *
 * How to create a CloudFront cache policy:
 * https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/controlling-the-cache-key.html
 * How to create a CloudFront origin request policy:
 * https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/controlling-origin-requests.html
 */
const customHeaderName = 'OneLink-JS-Prerender';

exports.handler = (event, context, callback) => {
   const { Records: [{ cf: { request } }] } = event;
   const { headers, uri } = request;
   const { 'user-agent': [{ value: userAgent } = {}] = []} = headers;

   const userAgentRegex = new RegExp(/([Bb]ot|[Ss]pider|slurp|archiver)/);

   /*
     * This regex is for sub-folder deployments only. Make sure to include all lang/locale pairs.
     * Also, this regex is redundant in cases where CloudFront distributions are correctly configured 
     * with pattern matching behaviors: 
     * https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html?icmpid=docs_cf_help_panel#DownloadDistValuesPathPattern
     */
   const uriRegex       = new RegExp('^/({language folders bar seperated})');

   headers[customHeaderName.toLowerCase()] = [{
      key: customHeaderName,
      value: (uriRegex.test(uri) && userAgentRegex.test(userAgent)).toString()
   }];

   callback(null, request);

Origin Request Function:

'use strict';

/** Origin Request **/
const projectKey = '____-____-____-____';
const flattenKey = `P${projectKey}`;
// const TransPerfectDomain = 'www.onelink-edge.com'; // <- The current EC2 XAPIS
// const TransPerfectDomain = 'xapis.onelinkjs.com';  // <- The container XAPIS

exports.handler = (event, context, callback) => {
   const { Records: [{ cf: { request } }] } = event;
   const { headers, uri } = request;
   const {
      host: [{ value: host }],
      'onelink-js-prerender': [{ value: oneLinkJSPrerender }]
   } = headers;

   if (oneLinkJSPrerender === 'true') {
      request.querystring = `url=//${host}${uri}`;

      /* Set custom origin fields */
      request.origin = {
         custom: {
            domainName: TransPerfectDomain,
            port: 443,
            protocol: 'https',
            path: '',
            sslProtocols: ['TLSv1', 'TLSv1.1'],
            readTimeout: 5,
            keepaliveTimeout: 5,
            customHeaders: {}
         }
      };

      request.uri = `/xapis/Flatten/${flattenKey}`;
      headers.host = [{ key: 'host', value: TransPerfectDomain }];
   }

   callback(null, request);
};