Chev Chev - 25 days ago 15
Node.js Question

How do you stream a csv file into a node web app?

I just took on a new project at work. It's an internal ExpressJS application that allows the employees to input data. There is a feature that allows them to enter payroll adjustments but the page only allows them to enter a single adjustment at a time and hit submit. They want a bulk upload feature so they can upload a CSV file with potentially hundreds of adjustments.

I have the feature working but the file upload middleware I'm using places the file in a temporary directory. I'm currently reading in the file, parsing out the adjustment items, and applying the adjustments to the payroll database before deleting the temporary file from the server. It all works great but I really hate that I'm doing file system I/O as well as reading the file into memory when it's time to parse the data.

As I learn more about transporting data over http I'm realizing that it should be possible to stream the file in and parse the items as they come. If so, is there a popular module that I should use for this purpose? What are the easiest ways to go about streaming a file up to the server and parsing it on the fly?

Answer

https://www.npmjs.com/package/busboy https://www.npmjs.com/package/fast-csv

var Busboy = require('busboy'); // to handle the form
var csv = require('fast-csv');

function (req, res) {
  var busboy = new Busboy({ headers: req.headers });
  busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
    file.pipe(csv())
      .on('data', function (data) {
        console.log('YAY, just the data I wanted!', data);
      });
  });
  busboy.on('finish', function() {
    console.log('Done parsing form!');
    res.end();
  });
  req.pipe(busboy);
}
Comments