user1239798 user1239798 - 5 months ago 23
Python Question

how enable requests async mode?

for this code:

import sys

import gevent
from gevent import monkey

monkey.patch_all()

import requests
import urllib2

def worker(url, use_urllib2=False):
if use_urllib2:
content = urllib2.urlopen(url).read().lower()
else:
content = requests.get(url, prefetch=True).content.lower()
title = content.split('<title>')[1].split('</title>')[0].strip()

urls = ['http://www.mail.ru']*5

def by_requests():
jobs = [gevent.spawn(worker, url) for url in urls]
gevent.joinall(jobs)

def by_urllib2():
jobs = [gevent.spawn(worker, url, True) for url in urls]
gevent.joinall(jobs)

if __name__=='__main__':
from timeit import Timer
t = Timer(stmt="by_requests()", setup="from __main__ import by_requests")
print 'by requests: %s seconds'%t.timeit(number=3)
t = Timer(stmt="by_urllib2()", setup="from __main__ import by_urllib2")
print 'by urllib2: %s seconds'%t.timeit(number=3)
sys.exit(0)


this result:

by requests: 18.3397213892 seconds
by urllib2: 2.48605842363 seconds


in sniffer it looks this:


description: first 5 requests are sended by requests library, next 5 requests are sended by urllib2 library.
red - is time when work was freeze, dark - when data receiving... wtf?!

How it posible if socket library patched and libraries must work identically?
How use requests without requests.async for asynchronious work?

Answer

Sorry Kenneth Reitz. His library is wonderful.

I am stupid. I need select monkey patch for httplib like this:

gevent.monkey.patch_all(httplib=True)

Because patch for httplib is disabled by default.