Sam Odio Sam Odio - 7 months ago 74
JSON Question

What's the best way to load large JSON lists in Python?

I have access to a set of files (around 80-800mb each). Unfortunately, there's only one line in every file. The line contains exactly one JSON object (a list of lists). What's the best way to load and parse it into smaller JSON objects?

Answer Source

There is already a similar post here. Here is the solution they proposed:

import json
with open('file.json') as infile:
  o = json.load(infile)
  chunkSize = 1000
  for i in xrange(0, len(o), chunkSize):
    with open('file_' + str(i//chunkSize) + '.json', 'w') as outfile:
      json.dump(o[i:i+chunkSize], outfile)
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download