delavnog delavnog - 1 year ago 101
Python Question

Log all requests from the python-requests module

I am using python Requests. I need to debug some

OAuth
activity, and for that I would like it to log all requests being performed. I could get this information with
ngrep
, but unfortunately it is not possible to grep https connections (which are needed for
OAuth
)

How can I activate logging of all URLs (+ parameters) that
Requests
is accessing?

Answer Source

The underlying urllib3 library logs all new connections and URLs with the logging module, but not POST bodies. For GET requests this should be enough:

import logging

logging.basicConfig(level=logging.DEBUG)

which gives you the most verbose logging option; see the logging HOWTO for more details on how to configure logging levels and destinations.

Short demo:

>>> import requests
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> r = requests.get('http://httpbin.org/get?foo=bar&baz=python')
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /get?foo=bar&baz=python HTTP/1.1" 200 353

The following messages are logged:

  • INFO: New connections (HTTP or HTTPS)
  • INFO: Dropped connections
  • INFO: Redirects
  • WARN: Connection pool full (if this happens often increase the connection pool size)
  • WARN: Retrying the connection
  • DEBUG: Connection details: method, path, HTTP version, status code and response length