Matt Matt - 10 months ago 39
Javascript Question

Garbage collection can't keep up with Buffer creation and removal

I have a method that runs every 2 seconds to capture a video stream to canvas and write it to file:

function capture(streamName, callback) {
var buffer,

_ctx = _canvas[streamName].getContext('2d');
_ctx.drawImage(_video[streamName], 0, 0);
dataURL = _canvas[streamName].toDataURL('image/png');
dataSplit = dataURL.split(",")[1];
buffer = new Buffer(dataSplit, 'base64');

fs.writeFileSync(directory + streamName + '.png', buffer);

setInterval(function() {
// Called from here
gameState.pollForState(processId, activeScreens[currentScreenIndex], function() {
// do things...
}, 2000);

exists as a running
exists as a
. The method works, it just causes a memory leak.

The issue:

Garbage collection can't keep up with the amount of memory the method uses, memory leak ensues.

I have narrowed it down to this line:

buffer = new Buffer(dataSplit, 'base64');

If I comment that out, there is some accumulation of memory (~100MB) but it drops back down every 30s or so.

What I've tried:

Some posts suggested
buffer = null;
to remove the reference and mark for garbage collection, but that hasn't changed anything.

Any suggestions?


Allocation Profile:

Just to quantify. After about 30 minutes of run time it sits at 2 GB memory used. This is an Electron (chromium / desktop) app.

Pre-allocating the buffer is what fixed it. This means that in addition to scoping
outside of the function, you need to reuse the created buffer with
. In order to keep proper headers, make sure that you use the
parameter of

Answer Source

Matt, I am not sure what was not working with the pre-allocated buffers, so I've posted an algorithm of how such pre-allocated buffers could be used. The key thing here is that buffers are allocated only once for that reason there should not be any memory leak.

var buffers = [];
var bsize = 10000;

// allocate buffer pool
for(var i = 0; i < 10; i++ ){
    buffers.push({free:true, buf: new Buffer(bsize)});

// sample method that picks one of the buffers into use
function useOneBuffer(data){
    // find a free buffer
    var theBuf;
    var i = 10;
    while((typeof theBuf==='undefined')&& i < 10){
            theBuf = buffers[i];
    } = false;
    // start doing whatever you need with the buffer, write data in needed format to it first
    // BUT do not allocate
    // also, you may want to clear-write the existing data int he buffer, just in case before reuse or after the use.
    if(typeof theBuf==='undefined'){
        // return or throw... no free buffers left for now
    // .... continue using

    // dont forget to pass the reference to the buffers member along because 
    // when you are done, toy have to mark it as free, so that it could be used again
    // = true;

Did you try something like this? Where did it fail?