Utsav Utsav - 5 months ago 13
Node.js Question

Frequent file reads vs accessing huge Array

I have a list of 40000 words from which I want to frequently return 20-40 words at random using Javascript(node.js) at request from client-side. Would it be better to read it every time from a file or to store it once in an array and then access it?


Huge Array (∈ RAM):

If you have a good RAM, loading your file lines in Huge array is better . Just, don't forget to increase the use of RAM when running your Node.js app .

node --max_old_space_size=2000 index.js //#default is 512Mo

Then , in load file's lines into array:

var JFile=require('jfile');
var words=new JFile('words.txt');
//--> words.lines // return an array of lines, then , you can handle it
 var between20_40= Math.floor(Math.random() * (40 - 20 + 1)) + 20;
 var randome_words=words.lines.sort().slice(0,between20_40) // will get n lines randomly (n between 20 & 40)

Access file (∈ HDD ):

IF your RAM capacity is small & you worry about it, accessing file is better :

var spawn = require('child_process').spawn;
 var between20_40= Math.floor(Math.random() * (40 - 20 + 1)) + 20;
var shellSyntaxCommand = `sort -R words.txt | head -n ${between20_40}`;
var output=spawn('sh', ['-c', shellSyntaxCommand], { stdio: 'inherit' });

Conclusion :

Dealing with something in RAM is much better than something in HDD . Thus, if you have a good RAM capacity , we recommend the first option.