user1276919 user1276919 - 2 months ago 28x
JSON Question

What is the best way to include huge json into node.js script?

How to include huge json file (550MB) into node.js script?

I tried:

var json = require('./huge-data-set.json')

The script was executed with increased --max-old-space-size parameter

node --max-old-space-size=4096 diff.js

it failed with:

throw new Error('"toString()" failed');

Error: "toString()" failed
at Buffer.toString (buffer.js:500:11)
at Object.fs.readFileSync (fs.js:552:33)
at Object.Module._extensions..json (module.js:556:20)


The error arises because you're trying to create a (single) string that is too large.

Some possible solutions:

  • Use a different backing store (e.g. sqlite db) and query data as you need instead of loading it all into memory at once
  • If your data is an array at the top level, consider having each array element on a separate line, that way you can just buffer a line and JSON.parse() that, then continue to the next line. However, you could still run into memory/GC issues if the parsed JavaScript values eat up too much space.