brfox brfox - 8 months ago 38
Python Question

Confused by python's unicode regex errors

Can someone explain why the middle code excerpt in python 2.7x throws an error?

import re
walden = "Waldenström"

s1 = "ö"
s2 = "Wal"
s3 = "OOOOO"

out = re.sub(s1, s3, walden)

out = re.sub("W", "w", walden)

# I need this one to work
out = re.sub('W', u'w', walden)

out = re.sub(u'W', 'w', walden)

out = re.sub(s2, s1, walden)

I'm very confused and have tried reading the manual


walden is a str:

walden = "Waldenström"

This code replaces a character with a unicode string:

re.sub('W', u'w', walden)

The result of that should be u'w' + "aldenström". This is the part that fails.

In order to concatenate str and unicode, both have to be first converted to unicode. The result is unicode as well.

The problem is, the interpreter does not know how to convert 'ö' to unicode, because it does not know which encoding to use. The result is ambiguous.

The solution is to convert yourself before doing the replacement:

re.sub('W', u'w', unicode(walden, encoding))

The encoding should be the one you use to create that file, e.g.

re.sub('W', u'w', unicode(walden, 'utf-8'))