user2814599 user2814599 - 1 year ago 175
Node.js Question

Prerender with nginx and node.js returns 504

If I understand things correctly I can setup nginx in a way that it handles crawlers (instead of nodejs doing it). So I removed

app.use(require('prerender-node').set('prerenderToken', 'token'))
from express configuration and made the following nginx setup (I do not use prerender token):

# Proxy / load balance (if more than one node.js server used) traffic to our node.js instances
upstream my_server_upstream {
keepalive 64;

server {
listen 80;

access_log /var/log/nginx/test_access.log;
error_log /var/log/nginx/test_error.log;
root /var/www/client;

# Static content
location ~ ^/(components/|app/|bower_components/|assets/|robots.txt|humans.txt|favicon.ico) {
root /;
try_files /var/www/.tmp$uri /var/www/client$uri =404;
access_log off;
sendfile off;

# Route traffic to node.js for specific route: e.g. /
location ~ ^/(api/|user/|en/user/|ru/user/|auth/||sitemap.xml) {
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_pass_header X-CSRFToken;
sendfile off;
# Tells nginx to use the upstream server
proxy_pass http://my_server_upstream;

location / {
root /var/www/client;
index index.html;
try_files $uri @prerender;
access_log off;
sendfile off;

location @prerender {
set $prerender 0;
if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
set $prerender 1;
if ($args ~ "_escaped_fragment_") {
set $prerender 1;
if ($http_user_agent ~ "Prerender") {
set $prerender 0;

#resolve using Google's DNS server to force DNS resolution and prevent caching of IPs

if ($prerender = 1) {
#setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
set $prerender "";
rewrite .* /$scheme://$host$request_uri? break;
proxy_pass http://$prerender;
if ($prerender = 0) {
rewrite .* /index.html$is_args$args break;

But when I test it by
I get
got 504 in 344ms for

Node version is 6.9.1. I use vagrant to setup environment.

Answer Source

The above configuration works fine. All it was missing is an entry in /etc/hosts :

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download