thehydrogen thehydrogen - 1 month ago 9
HTML Question

Error in open.connection(x,"rb") : HTTP error 406

I am trying to read the contents of a website using

read_html
in R. However, for some websites like
http://benchmarkrealestate.com/
, I get this error.
Error in open.connection(x,"rb") : HTTP error 406


What does this error mean? This only happens in some websites. I tried to look it up online, but wasn't able to find the exact reason why I get this error.

How do I fix this?

Answer

406 Not Acceptable

The requested resource is capable of generating only content not acceptable according to the Accept headers sent in the request.

The sentence above is lifted right off of Wikipedia.

Basically, whenever a Web crawler makes a request to a website, it often identifies itself, its application type and other information by submitting a characteristic identification string to its operating peer, i.e. the web server. In this case, this identification is transmitted in a header field called User-Agent.

One way to have the content of the web page returned to your console is to set your user-agent information to something identifiable with the help of the curl package:

library(xml2)
library(rvest)
library(curl)

web_content <- read_html(curl('http://benchmarkrealestate.com/', handle = new_handle("useragent" = "Mozilla/5.0")))

You may also want to read up on header fields.

Comments