Adam Rackis Adam Rackis - 3 months ago 43
Javascript Question

gulp-foreach not looping over all files

I have a very basic usage of gulp-foreach that's failing so badly I imagine I must be doing something wrong; I'm just (for now) trying to print all files found in gulp.src. When doing this with

gulp-foreach
only a very, very small sample of the files are ever printed—about 15 out of many hundreds. When using gulp-print things work as expected.

const gulp = require('gulp'),
foreach = require('gulp-foreach'),
gulpPrint = require('gulp-print');

gulp.src([`../../**/*.js`].concat([]))
.pipe(gulpPrint(path => {
console.log(path);
}))
.pipe(foreach((stream, file) => {
console.log(file.path);
return stream;
}))


The code with the
foreach
is triggered about 15 times, interleaved with the
gulpPrint
, followed by a flood of
gulpPrint
statements alone. If I comment out the
gulpPrint
part,
foreach
prints about 15 times and then exits, and
gulpPrint
alone, as expected, floods the console with every file found, as it should.

I can just use
gulp-print
, but I'd like to know what I'm doing wrong with
gulp-foreach
.

Answer

First of all: gulp-foreach isn't really the right plugin for your use case.

gulp-foreach allows you to create a substream for each file and emit the results of the substream into the original stream. That means you can emit multiple files for each input file (see this example). In this regard it is very similar to the higher-order function flatmap in functional programming. Which is why the author of gulp-foreach has recently renamed the plugin to gulp-flatmap.

All you are interested in is to access each file in the stream and print its path, which makes this a case of the higher-order function map. The right plugins for this are map-stream or gulp-tap (which is what the author of gulp-foreach used to recommend).

Now to your actual question.

As already mentioned gulp-foreach emits all the files from the created substream to the original stream. However you are doing nothing with the files that are written to the original stream. Node.js object streams by default have a highWaterMark of 16, so as soon as gulp-foreach has written 16 files to the original stream the buffer is full and gulp-foreach blocks.

In order to fix this you need to consume the files that gulp-foreach writes to the stream. You could simply use a dummy .on('data') handler for this:

gulp.src([`../../**/*.js`].concat([]))
  .pipe(gulpPrint(path => {
    console.log(path);
  }))
  .pipe(foreach((stream, file) => {
    console.log(file.path);
    return stream;
  }))
  .on('data', (data) => { });

However the usual way to do this in gulp is to let gulp consume the stream instead by simply returning the stream from the task it is created in:

gulp.task('default', function() {
  return gulp.src([`../../**/*.js`].concat([]))
    .pipe(gulpPrint(path => {
       console.log(path);
    }))
    .pipe(foreach((stream, file) => {
       console.log(file.path);
       return stream;
    }));
});
Comments