Abhishek Shivkumar Abhishek Shivkumar - 11 months ago 48
HTTP Question

How to send a repeat request in urllib2.urlopen in Python if the first call is just stuck?

I am making a call to a URL in Python using urllib2.urlopen in a while(True) loop

My URL keeps changing every time (as there is a change in a particular parameter of the URL every time).

My code look as as follows:

def get_url(url):
'''Get json page data using a specified API url'''
response = urlopen(url)
data = str(response.read().decode('utf-8'))
page = json.loads(data)
return page

I am calling the above method from the main function by changing the url every time I make the call.

What I observe is that after few calls to the function, suddenly (I don;t know why), the code gets stuck at the statement

response = urlopen(url)

and it just waits and waits...

How do I best handle this situation?

I want to make sure that if it does not respond within say 10 seconds, I make the same call again.

I read about

response = urlopen(url, timeout=10)

but then what about the repeated call if this fails?

Answer Source

Depending on how many retries you want to attempt, use a try/catch inside a loop:

while True:
        response = urlopen(url, timeout=10)
        # do something with the error
# do something with response
data = str(response.read().decode('utf-8'))

This will silence all exceptions, which may not be ideal (more on that here: Handling urllib2's timeout? - Python)