Tomer Zait Tomer Zait - 2 months ago 23
Python Question

Socks proxy in tornado AsyncHttpClient

How can I use socks proxy in tornado AsyncHttpClient?
I found it possible to use only HTTP Proxy without changing the lib...

Answer

According to the documentation, proxy support is only available for the libcurl implementation of AsyncHTTPClient.

If you will take a deeper look at the HTTPRequest object you're passing to the fetch() method, you'll notice there's an extra prepare_curl_callback argument, which can call setopt on the PyCurl object before the request is sent.

Here's a little example of such prepare_curl_callback function:

import pycurl

def prepare_curl_socks5(curl):
    curl.setopt(pycurl.PROXYTYPE, pycurl.PROXYTYPE_SOCKS5)

And a full usage example:

import tornado
import tornado.ioloop
import tornado.gen
import tornado.httpclient
import pycurl

def prepare_curl_socks5(curl):
    curl.setopt(pycurl.PROXYTYPE, pycurl.PROXYTYPE_SOCKS5)

@tornado.gen.coroutine
def main():
    # set CurlAsyncHTTPClient the default AsyncHTTPClient
    tornado.httpclient.AsyncHTTPClient.configure(
        "tornado.curl_httpclient.CurlAsyncHTTPClient")

    http_client = tornado.httpclient.AsyncHTTPClient()
    http_request = tornado.httpclient.HTTPRequest(
        "http://jsonip.com",
        prepare_curl_callback=prepare_curl_socks5,
        proxy_host="localhost",
        proxy_port=9050
    )
    response = yield http_client.fetch(http_request)

    print response.body

if __name__ == '__main__':
    tornado.ioloop.IOLoop.instance().run_sync(main)

The additional keyword argument prepare_curl_callback=prepare_curl_socks5 to the fetch() call does the magic, making CURL use SOCKS5 proxy instead of the default - HTTP proxy.

Comments