punkish punkish - 2 months ago 14
Node.js Question

nodejs/Mac and open files limit

I have written a program that recursively reads files in one directory, modifies them and writes them into another directory. Everytime I run that program, it croakes after a couple of hundred iterations. I just run it again, and it seems to complete the task.

Either nodejs or Mac OS X or, most likely, nodejs-on-Mac-OS-X, seems to have some kind of limit on the number of files that can be opened at one time. Searching around, I see that a solution is to use something like

ulimit -n 10480
and all will be well. Is that the right way? Instinctively, I'd rather not tinker with my system settings and rather modify my program to work within the limits.

An observation: Earlier I used to use Perl to do the task I've described above, and I never had a problem. I am assuming it was because I was opening, transforming, then closing the file, and then moving along. In nodejs, using
async
mode, I have no way of closing a file before going on to the next file. If I do the task in
sync
mode, it works fine.

Answer Source

You can use the async library with the limit commands to limit the number of files processed to a certain number. For example :

async.eachLimit(files, 1000, function (file, next) {
    processFile(file, next);
}, done);

If you wish to process a single file before going to the next one just use eachSeries.

async.eachSeries(files, function (file, next) {
    processFile(file, next);
}, done);