Lucas Ou Lucas Ou - 2 months ago 7
Python Question

django serving robots.txt efficiently

Here is my current method of serving robots.txt

url(r'^robots\.txt/$', TemplateView.as_view(template_name='robots.txt',
content_type='text/plain')),


I don't think that this is the best way. I think it would be better if it were just a pure
static resource and served statically. But the way my django app is structured is that the static root and all subsequent static files are located in

http://my.domain.com/static/stuff-here


Any thoughts? I'm amateur at django but

TemplateView.as_view(template_name='robots.txt',
content_type='text/plain')


looks a lot more resource consuming than just a static call to my static directory which is served on nginx.

Answer

Yes, robots.txt should not be served by Django if the file is static. Try something like this in your Nginx config file:

location  /robots.txt {
    alias  /path/to/static/robots.txt;
}

See here for more info: http://wiki.nginx.org/HttpCoreModule#alias

Same thing applies to the favicon.ico file if You have one.