I'm pulling data out of a Google doc, processing it, and writing it to a file (that eventually I will paste into a Wordpress page).
It has some non-ASCII symbols. How can I convert these safely to symbols that can be used in HTML source?
Currently I'm converting everything to Unicode on the way in, joining it all together in a Python string, then doing:
f = codecs.open('out.txt', mode="w", encoding="iso-8859-1")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position
12286: ordinal not in range(128)
row = [unicode(x.strip()) if x is not None else u'' for x in row]
all_html = row + "<br/>" + row
f = open('out.txt', 'w')
Deal exclusively with unicode objects as much as possible by decoding things to unicode objects when you first get them and encoding them as necessary on the way out.
If your string is actually a unicode object, you'll need to convert it to a unicode-encoded string object before writing it to a file:
foo = u'Δ, Й, ק, م, ๗, あ, 叶, 葉, and 말.' f = open('test', 'w') f.write(foo.encode('utf8')) f.close()
When you read that file again, you'll get a unicode-encoded string that you can decode to a unicode object:
f = file('test', 'r') print f.read().decode('utf8')