crashfocus crashfocus - 1 month ago 15
JSON Question

Read from large MongoDB, write to JSON

I have a MongoDB with about 70gb of data in it. I need to download the entire DB into one JSON document, preferably with Python. However, when I try and do

json.dump(object, file_object)
a couple million times, I tend to unintentionally create a malformed JSON. Storing the entire DB in memory isn't an option, because, well, 70gb > 16gb of RAM. How can I write JSON documents, one at a time, to the same file object in Python?

Thanks!

Answer

Use mongoexport to export the data:

mongoexport --db test --collection traffic --out traffic.json