user2814599 user2814599 - 1 month ago 16
Node.js Question

Prerender with nginx and node.js returns 504

If I understand things correctly I can setup nginx in a way that it handles crawlers (instead of nodejs doing it). So I removed

app.use(require('prerender-node').set('prerenderToken', 'token'))
from express configuration and made the following nginx setup (I do not use prerender token):

# Proxy / load balance (if more than one node.js server used) traffic to our node.js instances
upstream my_server_upstream {
server 127.0.0.1:9000;
keepalive 64;
}

server {
listen 80;

server_name test.local.io;
access_log /var/log/nginx/test_access.log;
error_log /var/log/nginx/test_error.log;
root /var/www/client;

# Static content
location ~ ^/(components/|app/|bower_components/|assets/|robots.txt|humans.txt|favicon.ico) {
root /;
try_files /var/www/.tmp$uri /var/www/client$uri =404;
access_log off;
sendfile off;
}

# Route traffic to node.js for specific route: e.g. /socket.io-client
location ~ ^/(api/|user/|en/user/|ru/user/|auth/|socket.io-client/|sitemap.xml) {
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_pass_header X-CSRFToken;
sendfile off;
# Tells nginx to use the upstream server
proxy_pass http://my_server_upstream;
}

location / {
root /var/www/client;
index index.html;
try_files $uri @prerender;
access_log off;
sendfile off;
}

location @prerender {
set $prerender 0;
if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
set $prerender 1;
}
if ($args ~ "_escaped_fragment_") {
set $prerender 1;
}
if ($http_user_agent ~ "Prerender") {
set $prerender 0;
}

#resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
resolver 8.8.8.8;

if ($prerender = 1) {
#setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
set $prerender "127.0.0.1:3000";
rewrite .* /$scheme://$host$request_uri? break;
proxy_pass http://$prerender;
}
if ($prerender = 0) {
rewrite .* /index.html$is_args$args break;
}
}
}


But when I test it by
curl test.local.io?_escaped_fragment_=
I get
got 504 in 344ms for http://test.local.io


Node version is 6.9.1. I use vagrant to setup environment.

Answer

The above configuration works fine. All it was missing is an entry in /etc/hosts : 127.0.0.1 test.local.io