Shane Shane - 10 days ago 6
HTTP Question

What should I do with hanging http POST request in python?

Sample code:

socket.setdefaulttimeout(150)

MechBrowser = mechanize.Browser()
Header = {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 GTB7.1 (.NET CLR 3.5.30729)'}
Url = "http://example.com"
Data = "Justatest=whatever&letstry=doit"
Request = urllib2.Request(Url, Data, Header)
Response = MechBrowser.open(Request)
Response.close()


I don't think there's anything wrong with above codes, but every now and then I'll get hanging http POST request which prevents the whole script from continuously running. I already used
socket.setdefaulttimeout(150)
how come it's not working? What is the reason causing this problem? And what should I do to get rid of this?

Answer

Found the problem.

I've been using requests a lot recently and realized that the timeout you set in both mechanize and requests are "NOT a time limit on the entire response download", which means if the connection is really slow and there are still data transferring, it will not timeout, which could hang the connection for quite a long time.

So what I've done is wrap those requests with threads and set timeout for those threads, and this way timeout is more accurate, just make sure you clean up/garbage collect those hanging connections.