I'm writing a webcrawler using Python and enjoying it a lot! But I've noticed some differences between the result produced by
I know this is an old question but maybe the answer can be still useful.
I had the same problem and what I did to solve it was creating a php file which printed the request headers. Then I executed a
curl and an
urlopen and I checked the differences between the headers. You can find an example of that script in PHP docs.
In addiction, you can go to your browser and check which headers are being send. Doing this I saw that
connection: close instead of
So finally I add the
keep-alive header and
urlopen began to work as
curl. This was my concrete problem but maybe yours is different due to the server requirements and you need to add or remove another header.