Dimitrios Dimitrios - 7 months ago 37
Python Question

Python 2.7 JSON dump UnicodeEncodeError

I have a file where each line is a json object like so:

{"name": "John", ...}

{...}


I am trying to create a new file with the same objects, but with certain properties removed from all of them.

When I do this, I get a UnicodeEncodeError. Strangely, If I instead loop over
range(n)
(for some number n) and use
infile.next()
, it works just as I want it to.

Why so? How do I get this to work by iterating over
infile
? I tried using
dumps()
instead of
dump()
, but that just makes a bunch of empty lines in the
outfile
.

with open(filename, 'r') as infile:
with open('_{}'.format(filename), 'w') as outfile:
for comment in infile:
decodedComment = json.loads(comment)
for prop in propsToRemove:
# use pop to avoid exception handling
decodedComment.pop(prop, None)
json.dump(decodedComment, outfile, ensure_ascii = False)
outfile.write('\n')


Here is the error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\U0001f47d' in position 1: ordinal not in range(128)


Thanks for the help!

Answer

The problem you are facing is that the standard file.write() function (called by the json.dump() function) does not support unicode strings. From the error message, it turns out that your string contains the UTF character \U0001f47d (which turns out to code for the character EXTRATERRESTRIAL ALIEN, who knew?), and possibly other UTF characters. To handle these characters, either you can encode them into an ASCII encoding (they'll show up in your output file as \XXXXXX), or you need to use a file writer that can handle unicode.

To do the first option, replace your writing line with this line:

json.dump(unicode(decodedComment), outfile, ensure_ascii = False)

The second option is likely more what you want, and an easy option is to use the codecs module. Import it, and change your second line to:

with codecs.open('_{}'.format(filename), 'w', encoding="utf-8") as outfile:

Then, you'll be able to save the special characters in their original form.