Coop Coop - 6 months ago 109
Javascript Question

Detect social bots in Node Express

I'm trying to detect for either of the following 2 options:


  • A specific list of bots (FacebookExternalHit|LinkedInBot|TwitterBot|Baiduspider)

  • Any bots that don't support the Crawable Ajax Specification



I've seen similar questions (How to recognize Facebook User-Agent) but nothing that explains how to do this in Node and Express.

I need to do this in a format like this:

app.get("*", function(req, res){
if (is one of the bots) //serve snapshot
if (is not one of the bots) res.sendFile(__dirname + "/public/index.html");
});

Answer

What you can do is use the request.headers object to check if the incoming request contains any UA information specific to that bot. A simple example.

Node

var http = require('http');

var server = http.createServer(function(req, res){

    if(req.headers['user-agent'] === 'facebookexternalhit/1.1') /* do something for the Facebook bot */


});

server.listen(8080);

Express

var http = require('http');
var express = require('express');
var app = express();

app.get('/', function(req, res){

    if(req.headers['user-agent'] === 'facebookexternalhit/1.1') /* do something for the Facebook bot */


});

app.listen(8080);
Comments