eran eran - 1 month ago 20
Python Question

With reading hebrew in urllib2 python

I read a webpage which contains hebrew characters, using:

response = ('').join(opener.open(url).readlines())


The result I get is mixed, some of the characters come back as unicode, which I can handle.

Some of the response seems garbled. In a format I cant recognize.
An example of the recieved text is:
שלך

More precisely, it looks like this (only a snippet...):

<h3 class="_52r al aps">About ‎גדי‎</h3><div>&#x5e9;&#x5dc;&#x5da; ....</div>


The text between the divs seems scrambled. Can I convert it to unicode?

Answer

You are looking at HTML entities; use the HTMLParser library to decode these:

>>> import HTMLParser
>>> h = HTMLParser.HTMLParser()
>>> print h.unescape('&#x5e9;&#x5dc;&#x5da;')
שלך
>>> h.unescape('&#x5e9;&#x5dc;&#x5da;')
u'\u05e9\u05dc\u05da'

To read a full urllib2 response, just use .read():

response = opener.open(url).read()