Maxwell Bottiger Maxwell Bottiger - 1 year ago 87
JSON Question

Python dict/list mix is not JSON serializable

I'm a bit stumped. I'm trying to build JSON string, and it's generating a traceback in my program. The really odd thing, is that if I cut and paste into a console, the whole thing works.

First, here's the code.

def readgs2JSON(self, msg):
d = {"Channel" : "Readings"}
d["Sensor"] = msg.ReadingsChn.ReadingReport.attrib["Sensor"]
d["ReadingID"] = msg.ReadingsChn.ReadingReport.attrib["ReadingID"]
d["Detect"] = msg.ReadingsChn.ReadingReport.Data.attrib["Detect"]
d["Level"] = msg.ReadingsChn.ReadingReport.Data.attrib["Level"]
d["Units"] = msg.ReadingsChn.ReadingReport.Data.attrib["Units"]
if "Id" in msg.ReadingsChn.ReadingReport.Data.attrib:
d["Id"] = msg.ReadingsChn.ReadingReport.Data.attrib["Id"]
d["SUD"] = [el.attrib for el in msg.ReadingsChn.ReadingReport.Data.iterchildren()]
print d
return d


The variable msg is an objectified element produced by lxml. The dictionary, d generated by the code looks like this when you print it. (Sorry that's hard to read. It doesn't seem to want to line wrap nicely.)

{'Detect': 'NONE', 'Level': '0', 'SUD': [{'Type': 'int', 'Name':'Reading1', 'Value': '75856'}, {'Type': 'int', 'Name': 'Reading2', 'Value': '75857'}, {'Type': 'int', 'Name': 'Reading3', 'Value': '75858'}, {'Type': 'int', 'Name': 'Reading4', 'Value': '75859'}, {'Type': 'int', 'Name': 'ClockTicks', 'Value': '389'}, {'Type': 'array', 'Name': 'Spectrum', 'Value': 'None'}], 'Units': 'bars', 'ReadingID': 'R000009233', 'Sensor': 'SC001', 'Channel': 'Readings'}


So, when I execute json.dumps(d) inside my program, I get a traceback:

[Failure instance: Traceback: <type 'exceptions.TypeError'>: {'Type': 'int', 'Name': 'Reading1', 'Value': '75856'} is not JSON serializable
/usr/lib64/python2.7/site-packages/twisted/internet/base.py:1195:run
/usr/lib64/python2.7/site-packages/twisted/internet/base.py:1204:mainLoop
/usr/lib64/python2.7/site-packages/twisted/internet
/base.py:825:runUntilCurrent
/usr/lib64/python2.7/site-packages/twisted/internet/task.py:239:__call__
--- <exception caught here> ---
/usr/lib64/python2.7/site-packages/twisted/internet
/defer.py:149:maybeDeferred
/home/max/workspace/canary/CCSIEventHandler.py:26:tick
/home/max/workspace/canary/CCSIEventHandler.py:99:event2Msg
/home/max/workspace/canary/sensorcache.py:715:writeToBuffer
/home/max/workspace/canary/CCSI2JSON.py:37:pushCCSIMessage
/usr/lib64/python2.7/json/__init__.py:244:dumps
/usr/lib64/python2.7/json/encoder.py:207:encode
/usr/lib64/python2.7/json/encoder.py:270:iterencode
/usr/lib64/python2.7/json/encoder.py:184:default
]


So, the really odd thing, is if I copy the printed dict from my terminal, and past it into a python console, then the whole thing works!

>>> json.dumps(d)
'{"SUD": [{"Type": "int", "Name": "Reading1", "Value": "75856"}, {"Type": "int", "Name": "Reading2", "Value": "75857"}, {"Type": "int", "Name": "Reading3", "Value": "75858"}, {"Type": "int", "Name": "Reading4", "Value": "75859"}, {"Type": "int", "Name": "ClockTicks", "Value": "389"}, {"Type": "array", "Name": "Spectrum", "Value": "None"}], "Level": "0", "Detect": "NONE", "Units": "bars", "ReadingID": "R000009233", "Sensor": "SC001", "Channel": "Readings"}'


For the life of me, I can't spot the difference, but I'm no JSON expert. Has anyone else had this problem?

Turns out I was trying to serialize an lxml.etree._Attrib, which I thought was a dictionary, as it works like a dict. Here's the adjusted code that builds a dictionary, and works correctly.

def readgs2JSON(self, msg):
d = {"Channel" : "Readings"}
d["Sensor"] = msg.ReadingsChn.ReadingReport.attrib["Sensor"]
d["ReadingID"] = msg.ReadingsChn.ReadingReport.attrib["ReadingID"]
d["Detect"] = msg.ReadingsChn.ReadingReport.Data.attrib["Detect"]
d["Level"] = msg.ReadingsChn.ReadingReport.Data.attrib["Level"]
d["Units"] = msg.ReadingsChn.ReadingReport.Data.attrib["Units"]
if "Id" in msg.ReadingsChn.ReadingReport.Data.attrib:
d["Id"] = msg.ReadingsChn.ReadingReport.Data.attrib["Id"]
sud_list = []
for el in msg.ReadingsChn.ReadingReport.Data.iterchildren():
sud_dict = {}
for item in el.attrib:
sud_dict[item] = el.attrib[item]
sud_list.append(sud_dict)
if sud_list is not []:
d["SUD"] = sud_list
return d

Answer Source

Children of msg.ReadingsChn.ReadingReport.Data are of some special type, and they are not JSON serializable. The reason it is printed to the console correctly is because that type has overridden __str__ or __repr__.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download