qweruiop qweruiop - 1 year ago 226
Python Question

Python requests decode using a different charset

I'm using

to fetch a webpage but the automatically inferred encoding in
is incorrect. I need to decode the bytes in unicode, but I didn't find an API for this.
is Unicode but not the byte array. Any idea?

Answer Source

From the documentation (emphasis added):

When you make a request, Requests makes educated guesses about the encoding of the response based on the HTTP headers. The text encoding guessed by Requests is used when you access r.text. You can find out what encoding Requests is using, and change it, using the r.encoding property:

>>> r.encoding
>>> r.encoding = 'ISO-8859-1'

If you change the encoding, Requests will use the new value of r.encoding whenever you call r.text. You might want to do this in any situation where you can apply special logic to work out what the encoding of the content will be. For example, HTTP and XML have the ability to specify their encoding in their body. In situations like this, you should use r.content to find the encoding, and then set r.encoding. This will let you use r.text with the correct encoding.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download