Phil Sheard Phil Sheard - 5 months ago 11
Python Question

TypeError when calling requests_oauthlib multiple times (error actually raised within urllib3)

I'm writing a script to pull some data from the Twitter API. It's use of OAuth 1.1 means I'm using the

requests_oauthlib
helper library on top of
requests
to authenticate the session.

The first call to the API works, but then subsequent calls give a
TypeError
as follows:

/Users/phil/code/Virtualenv/req_test/lib/python2.7/site-packages/requests/packages/urllib3/connection.pyc in __init__(self, *args, **kw)
124
125 # Superclass also sets self.source_address in Python 2.7+.
--> 126 _HTTPConnection.__init__(self, *args, **kw)
127
128 def _new_conn(self):

TypeError: unbound method __init__() must be called with HTTPConnection instance as first argument (got VerifiedHTTPSConnection instance instead)


Looks like there's something persisting in the session as it's always on repeated use that the error comes. I've tried a clean virtualenv with latest versions installed via pip and no difference.

I'm using the context manager approach so thought that the session would be destroyed after each call, preventing this from happening:

with ro.OAuth1Session(**self._auth) as s:
response = s.get(url)


Any fix or pointers to understand what's causing the problem would be appreciated.

Edit: I've tried a different approach, using the alternative way of building a session as described on the
requests
docs (http://docs.python-requests.org/en/master/user/authentication/) but same error is raised.

Edit: Full stack in case it's useful:

/Users/phil/code/Virtualenv/req_test/lib/python2.7/site-packages/requests/sessions.pyc in get(self, url, **kwargs)
485
486 kwargs.setdefault('allow_redirects', True)
--> 487 return self.request('GET', url, **kwargs)
488
489 def options(self, url, **kwargs):

/Users/phil/code/Virtualenv/req_test/lib/python2.7/site-packages/requests/sessions.pyc in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
473 }
474 send_kwargs.update(settings)
--> 475 resp = self.send(prep, **send_kwargs)
476
477 return resp

/Users/phil/code/Virtualenv/req_test/lib/python2.7/site-packages/requests/sessions.pyc in send(self, request, **kwargs)
583
584 # Send the request
--> 585 r = adapter.send(request, **kwargs)
586
587 # Total elapsed time of the request (approximately)

/Users/phil/code/Virtualenv/req_test/lib/python2.7/site-packages/requests/adapters.pyc in send(self, request, stream, timeout, verify, cert, proxies)
401 decode_content=False,
402 retries=self.max_retries,
--> 403 timeout=timeout
404 )
405

/Users/phil/code/Virtualenv/req_test/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.pyc in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, **response_kw)
564 # Request a connection from the queue.
565 timeout_obj = self._get_timeout(timeout)
--> 566 conn = self._get_conn(timeout=pool_timeout)
567
568 conn.timeout = timeout_obj.connect_timeout

/Users/phil/code/Virtualenv/req_test/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.pyc in _get_conn(self, timeout)
254 conn = None
255
--> 256 return conn or self._new_conn()
257
258 def _put_conn(self, conn):

/Users/phil/code/Virtualenv/req_test/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.pyc in _new_conn(self)
800 conn = self.ConnectionCls(host=actual_host, port=actual_port,
801 timeout=self.timeout.connect_timeout,
--> 802 strict=self.strict, **self.conn_kw)
803
804 return self._prepare_conn(conn)

/Users/phil/code/Virtualenv/req_test/lib/python2.7/site-packages/requests/packages/urllib3/connection.pyc in __init__(self, host, port, key_file, cert_file, strict, timeout, **kw)
208
209 HTTPConnection.__init__(self, host, port, strict=strict,
--> 210 timeout=timeout, **kw)
211
212 self.key_file = key_file

/Users/phil/code/Virtualenv/req_test/lib/python2.7/site-packages/requests/packages/urllib3/connection.pyc in __init__(self, *args, **kw)
124
125 # Superclass also sets self.source_address in Python 2.7+.
--> 126 _HTTPConnection.__init__(self, *args, **kw)
127
128 def _new_conn(self):

TypeError: unbound method __init__() must be called with HTTPConnection instance as first argument (got VerifiedHTTPSConnection instance instead)

Answer

Edit: Also added reference to reloading container library within IPython / Jupyter.

After a fair bit of reading, it appears the problem may arise when you call get on a request but don't access the body of the response (which is what I did whilst building / debugging the flow):

http://docs.python-requests.org/en/master/user/advanced/#keep-alive

"Note that connections are only released back to the pool for reuse once all body data has been read; be sure to either set stream to False or read the content property of the Response object."

So the answer is to make sure the first thing you do after making a request is flush the response by calling Response.content, Response.json() or similar method.

The problem has only arisen when using the requests_oauthlib library because the session it uses is less common. I've done similar with the Facebook and LinkedIn APIs without issue, by using query parameters that don't affect the session object itself.

It also arose most often when reloading the helper library within IPython / Jupyter. Quitting the notebook or command line session then restarting would remove the issue.