B T B T - 1 month ago 13
HTTP Question

Is it possible to tell web crawlers two domains point to the same place without using a 301 redirect?

I have two domains that point to the same sever and I'd like to keep it so that a user can go to either of these URLs and the URL will remain as typed in the browser's address bar ("redirecting" would be ok as long as browsers keep the original url visible). But I've read that this can be bad for SEO.

Is there some way to tell seach engines that two domains point to the same place (perhaps in robots.txt tho that's not looking promising). Basically I'd like a 301 redirect used for just web crawlers.

I could try detecting when an http request is coming from a crawler and redirecting them, but I haven't seen a standard way to do this.

Answer

You can use a link element in the HTML head with a rel attribute of "canonical", which then will indicate to web browsers that the "real" domain is whatever you set the href attribute to.

Example:

  <link rel="canonical" href="http://www.example.com/your-other-site-with-identical-content">

Source: https://webmasters.googleblog.com/2009/02/specify-your-canonical.html

The fact that google has supported this for such a long time (from 2009 to present) is a strong indicator that this may be the way to go.