jony jony - 1 month ago 16
Python Question

Decoding urllib.request response

I'm getting this response when I open this url:

r = Request(r'http://airdates.tv/')
h = urlopen(r).readline()
print(h)


Response:

b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00\xed\xbdkv\xdbH\x96.\xfa\xbbj\x14Q\xaeuJ\xce\xee4E\x82\xa4(9m\xe7\xd2\xd3VZ\xaf2e\xab2k\xf5\xc2\n'


What encoding is this?
Is there a way to decode it based on the standard library?

Thank you in advance for any insight on this matter!

PS: It seems to be gzip.

Answer

It's gzip compressed HTML, as you suspected.

Rather than use urllib use requests which will decompress the response for you:

import requests

r = requests.get('http://airdates.tv/')
print(r.text)

You can install it with pip install requests, and never look back.


If you really must restrict yourself to the standard library, then decompress it with the gzip module:

import gzip
import urllib2
from cStringIO import StringIO

f = urllib2.urlopen('http://airdates.tv/')

# how to determine the content encoding
content_encoding = f.headers.get('Content-Encoding')
print(content_encoding)

# how to decompress gzip data
if content_encoding == 'gzip':
    gz = gzip.GzipFile(fileobj=StringIO(f.read())
    response = gz.read()