CentAu CentAu - 6 months ago 9
Python Question

Python, write json / dictionary objects to a file iteratively (one at a time)

I have a large

for loop
in which I create json objects and I would like to be able to stream write the object in each iteration to a file. I would like to be able to use the file later in a similar fashion later (read objects one at a time).
My json objects contain newlines and I can't just dump each object as a line in a file.
How can I achieve this?

To make it more concrete, consider the following:

for _id in collection:
dict_obj = build_dict(_id) # build a dictionary object
with open('file.json', 'a') as f:
stream_dump(dict_obj, f)

is the function that I want.

Note that I don't want to create a large list and dump the whole list using something like
json.dump(obj, file)
. I want to be able to append the object to the file in each iteration.



You need to work with a subclass of JSONEncoder and then proxy the build_dict function

from __future__ import (absolute_import, division, print_function,)
#                        unicode_literals)

import collections
import json

mycollection = [1, 2, 3, 4]

def build_dict(_id):
    d = dict()
    d['my_' + str(_id)] = _id
    return d

class SeqProxy(collections.Sequence):
    def __init__(self, func, coll, *args, **kwargs):
        super(SeqProxy, *args, **kwargs)

        self.func = func
        self.coll = coll

    def __len__(self):
        return len(self.coll)

    def __getitem__(self, key):
        return self.func(self.coll[key])

class JsonEncoderProxy(json.JSONEncoder):
    def default(self, o):
            iterable = iter(o)
        except TypeError:
            return list(iterable)
        # Let the base class default method raise the TypeError
        return json.JSONEncoder.default(self, o)

jsonencoder = JsonEncoderProxy()
collproxy = SeqProxy(build_dict, mycollection)

for chunk in jsonencoder.iterencode(collproxy):



To read it back chunk by chunk you need to use JSONDecoder and pass a callable as object_hook. This hook will be called with each new decoded object (each dict in your list) when you call JSONDecoder.decode(json_string)