I have access to a set of files (around 80-800mb each). Unfortunately, there's only one line in every file. The line contains exactly one JSON object (a list of lists). What's the best way to load and parse it into smaller JSON objects?
There is already a similar post here. Here is the solution they proposed:
import json with open('file.json') as infile: o = json.load(infile) chunkSize = 1000 for i in xrange(0, len(o), chunkSize): with open('file_' + str(i//chunkSize) + '.json', 'w') as outfile: json.dump(o[i:i+chunkSize], outfile)