eran eran - 3 months ago 54
Python Question

With reading hebrew in urllib2 python

I read a webpage which contains hebrew characters, using:

response = ('').join(

The result I get is mixed, some of the characters come back as unicode, which I can handle.

Some of the response seems garbled. In a format I cant recognize.
An example of the recieved text is:

More precisely, it looks like this (only a snippet...):

<h3 class="_52r al aps">About ‎גדי‎</h3><div>&#x5e9;&#x5dc;&#x5da; ....</div>

The text between the divs seems scrambled. Can I convert it to unicode?


You are looking at HTML entities; use the HTMLParser library to decode these:

>>> import HTMLParser
>>> h = HTMLParser.HTMLParser()
>>> print h.unescape('&#x5e9;&#x5dc;&#x5da;')
>>> h.unescape('&#x5e9;&#x5dc;&#x5da;')

To read a full urllib2 response, just use .read():

response =