zeddex zeddex - 1 month ago 13
PHP Question

PHPBB Robots.txt Rules?

I am currently editing my Robots.txt which looks like this:

User-agent: *
Disallow: /adm/*
Disallow: /download/*
Disallow: /cache
Disallow: /files
Disallow: /viewforum.php?f=146
Disallow: /ucp.php
Disallow: /mcp.php
Disallow: /memberlist.php
Disallow: /config.php
Disallow: /cron.php
Disallow: /faq.php
Disallow: /report.php
Sitemap: http://www.website.com/app.php/sitemap.xml


I am wondering how to correctly do a few things however.

1) Would this correctly block search engines from accessing a forum area?

Disallow: /viewforum.php?f=146


I wanted one area hidden from search engines but the rest of the forum areas fully readable as normal.

2) How do you block access to the internal PHPBB folders and keep search engines out out admin? are these rules correct?

Disallow: /adm/*
Disallow: /download/*


3) Do the rules for php files work correctly?

Disallow: /ucp.php


Also is there anything else i should know or do?

Answer

You can only disallow directories via robots.txt, not files. First, remove the asterisks from the directories.

Disallow: /adm/
Disallow: /download/

As for the files you could add this to a < HEAD > section. It wont prevent the bot from fetching, but good bots should ignore it and move on.

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">