Hybrid Hybrid - 7 months ago 15
Python Question

Python 3.4.3 json.dumps() "is not JSON serializable" Error When Converting From Bytes

So I am having this issue where I am using urllib to GET a JSON response from a server, but when I try to convert the returned

b''
object into JSON, I get an error that says
"...is not JSON serializable"


Here is my Python code using urllib2 to make a GET request:

from urllib.request import Request, urlopen
from urllib.error import HTTPError
import json


hdr = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
'Accept': 'application/json',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Accept-Encoding': 'none',
'Accept-Language': 'en-US,en;q=0.8',
'Connection': 'keep-alive'}

file = Request('http://blahblahblah.com', headers=hdr)

try:
page = urlopen(file)
except HTTPError as e:
print(e)
print('')
return

content = page.read()
page.close()


print(page.info().get_content_charset()) # Returns 'utf-8'
print(content)
# print(json.dumps(content)) # Causes 'Not Serializable' Error
# print(content.decode('utf-8') # Causes 'UnicodeEncodeError' Error


Here is the response I am getting from the server:

b'{"game":{"id":1,"name":"Thief II: The Metal Age","slug":"thief-ii-the-metal-age","release_date":"2000-03-21","created_at":"2011-02-13 00:20:38 +0000","updated_at":"2016-03-15 19:41:25 +0000","alternative_names":[{"name":"Thief II: \xd0\xad\xd0\xbf\xd0\xbe\xd1\x85\xd0\xb0 \xd0\xbc\xd0\xb5\xd1\x82\xd0\xb0\xd0\xbb\xd0\xbb\xd0\xb0","comment":"Russian title"},{"name":"Dark Project II: The Metal Age","comment":"German title"},{"name":"Dark Project II: L\'Age de M\xc3\xa9tal","comment":"French title"}],"genres":[{"name":"Shooter"}],"themes":[{"name":"Action"},{"name":"Fantasy"},{"name":"Stealth"}],"rating":9.131189346313477,"release_dates":[{"platform_name":"Microsoft Windows","release_date":"2000-03-21"}],"companies":[{"id":4,"developer":false,"publisher":true,"name":"Eidos Interactive"},{"id":3,"developer":true,"publisher":false,"name":"Looking Glass Studios"},{"id":26,"developer":false,"publisher":true,"name":"Square Enix"}],"cover":{"url":"//res.cloudinary.com/igdb/image/upload/t_cover_small/qagoforxr6tofvpmgy9g.png","width":612,"height":650,"id":"qagoforxr6tofvpmgy9g"},"screenshots":[{"url":"//res.cloudinary.com/igdb/image/upload/t_screenshot_med/z0b9mqcqbtmnnxigekjc.jpg","title":"Microsoft Windows Title Screen","width":640,"height":480,"id":"z0b9mqcqbtmnnxigekjc"},{"url":"//res.cloudinary.com/igdb/image/upload/t_screenshot_med/puvydf5d6v0zirxfhzpg.jpg","title":"Microsoft Windows Ingame Screen","width":640,"height":480,"id":"puvydf5d6v0zirxfhzpg"}],"videos":[{"title":"Trailer","uid":"9C543B6uJ88"}]}}'


I have read elsewhere that I have to first convert the response to a
str
or use
utf-8
encoding, but when I try to replace
print(content)
with
print(content.decode('utf-8')
, I get the following error:

UnicodeEncodeError: 'charmap' codec can't encode characters in position 231-235: character maps to <undefined>

Answer

So I eventually figured it out - the problem was that my IDE (Pycharm 5.0) for some reason could not handle outputting utf-8 in the IDE console, which caused an error. The solution was to add the following line of code to the pycharm.exe.vmpoptions file in my windows computer:

-Dfile.encoding=UTF-8

Credit goes to another user for finding the same solution when they had a similar issue:

Source: Why unicode string is not shown on PyCharm's console?

Comments