crashfocus crashfocus - 7 months ago 50
JSON Question

Read from large MongoDB, write to JSON

I have a MongoDB with about 70gb of data in it. I need to download the entire DB into one JSON document, preferably with Python. However, when I try and do

json.dump(object, file_object)
a couple million times, I tend to unintentionally create a malformed JSON. Storing the entire DB in memory isn't an option, because, well, 70gb > 16gb of RAM. How can I write JSON documents, one at a time, to the same file object in Python?



Use mongoexport to export the data:

mongoexport --db test --collection traffic --out traffic.json