delavnog delavnog - 2 months ago 21
Python Question

Log all requests from the python-requests module

I am using python Requests. I need to debug some

OAuth
activity, and for that I would like it to log all requests being performed. I could get this information with
ngrep
, but unfortunately it is not possible to grep https connections (which are needed for
OAuth
)

How can I activate logging of all URLs (+ parameters) that
Requests
is accessing?

Answer

The underlying urllib3 library logs all new connections and URLs with the logging module, but not POST bodies. For GET requests this should be enough:

import logging

logging.basicConfig(level=logging.DEBUG)

which gives you the most verbose logging option; see the logging HOWTO for more details on how to configure logging levels and destinations.

Short demo:

>>> import requests
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> r = requests.get('http://httpbin.org/get?foo=bar&baz=python')
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /get?foo=bar&baz=python HTTP/1.1" 200 353

The following messages are logged:

  • INFO: New connections (HTTP or HTTPS)
  • INFO: Dropped connections
  • INFO: Redirects
  • WARN: Connection pool full (if this happens often increase the connection pool size)
  • WARN: Retrying the connection
  • DEBUG: Connection details: method, path, HTTP version, status code and response length