Sanich Sanich - 2 months ago 24
Python Question

Using unicode (Hebrew characters) with regular expression

I wrote script that find expressions in web page:

import sre, urllib2, sys, BaseHTTPServer
# -*- coding: utf-8 -*-
address = sys.argv[1]
web_handle = urllib2.urlopen(address)
website_text = website_handle.read()
matches = sre.findall(u"עברית", website_text)
for item in matches:
print iten


This script works if I use a "regular" regular expression (without Hebrew characters) and doesn't match anything if I use them.
What am I doing wrong?

edit
example:
url = https://en.wikipedia.org/wiki/Category:Countries

Answer

You need to ensure that the input string is also in UTF8 format.

Use unicode function with utf-8 as second argument:

website_text = unicode(website_text, "utf-8")

Everything should be in consistent encoding for unicode to work in Python 2.