DNorthrup DNorthrup - 1 year ago 59
Javascript Question

Node Script running out of memory (Loop within a loop)

I have a very basic script that I've been adding on functionality to bit by bit - Hardly optimized.

It's supposed to go through X

brands
and run an API call for each one (Each has a unique endpoint) and then save the results to a CSV.

The CSV portion worked perfectly, albeit was a tad slow. (~20 seconds for 200KB file).

However, when I went to wrap all of it within a loop for the
brands
it now times out. By default it'll run 2 minutes then give me an error
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory


Which I learned to append
--max_old_space_size=4000000
to give it excess memory (It also died at 2GB). At 4GB I didn't get the error but I was getting Windows memory leak warnings so it's VERY obvious my script is having issues. I feel like it's looping during loops, but unsure.

var http = require("https");
var qs = require("querystring");
var csvWriter = require('csv-write-stream');
var fs = require('fs');
var moment = require('moment');
let key = '';
var brands = ['REDACTED1','REDACTED2','REDACTED3','REDACTED4','REDACTED5','REDACTED6','REDACTED7','REDACTED8']
var dataRequest = function(token){
var date = new Date()-1; //Make it yesterday
var formattedDate = moment(date).format('YYYYMMDD');
var yesterdayDashes = moment(date).format('YYYY-MM-DD');
for (var k=0;k<=brands.length;k++){

var headers = [];
var csvBody = [];
var csvContent = "data:text/csv;charset=utf-8,";
var options = {
"method": "GET",
"hostname": "REDACTED",
"port": "443",
"path": "REDACTED/"+brands[k]+"/"+yesterdayDashes+"?REDACTED&access_token="+token,
"headers": {
"authentication": "Bearer "+token,
"content-type": "application/json",
"cache-control": "no-cache"
}
}

var req = http.request(options, function (res) {
var chunks = [];

res.on("data", function (chunk) {
chunks.push(chunk);
});

res.on("end", function () {
var body = Buffer.concat(chunks);
var object = JSON.parse(body);
var writer = csvWriter({
headers: ["REDACTED1", "REDACTED2", "REDACTED3", "REDACTED4", "REDACTED5", "REDACTED6", "REDACTED7", "REDACTED8", "REDACTED9"]
})
writer.pipe(fs.createWriteStream(brands[k] +'_' +formattedDate +'_DEMOGRAPHIC.csv'))
for (var i=0;i<object.length; i++){
writer.write([
object[i].field1.REDACTED1,
moment(object[i].optin.REDACTED2).format('YYYYMMDD HHMMSS'),
object[i].field2.REDACTED1,
object[i].field2.REDACTED2,
object[i].field2.REDACTED3,
object[i].field2.REDACTED4,
object[i].field2.REDACTED5,
object[i].field3.REDACTED6,
object[i].field3.REDACTED7
])
}
writer.end()
});
});
req.end();
}
}

var authToken = function(){

var form = qs.stringify({
grant_type: 'client_credentials',
client_credentials: 'client_id:client_secret',
client_id: 'cdg-trusted-client',
client_secret: 'REDACTED'
})
var options = {
"method": "POST",
"hostname": "REDACTED3",
"port": null,
"path": "REDACTED3",
"headers": {
"Content-Type": "application/x-www-form-urlencoded",
"cache-control": "no-cache"
}
};

var req = http.request(options, function (res) {
var chunks = [];

res.on("data", function (chunk) {
chunks.push(chunk);
});

res.on("end", function () {
var body = Buffer.concat(chunks);
var json = JSON.parse(body);
key = json['access_token'];
});
});

req.write(form);
req.end();
dataRequest(key);
}
authToken();


I removed sensitive information but all of the logic remains. This is a script I quickly threw together, but honestly going through it I don't really see any reason that it should require so much memory. I thought perhaps it was infinite looping but testing each loops within node directly I had no issues.

The flow starts getting the bearer token, once, then passing it to the function to pull data.

While I was debating putting this in CodeReview, this code isn't technically working at all.

Update
Modifying the first
for
loop now outputs an
unidentified
CSV immediately and doesn't completely anything else. However
console.log(brands[k])
is outputting the appropriate files.

Update 2

My version of JS debugging is putting
console.log()
everywhere, and I notice once I get below the
http.request
init, brands[k] suddenly becomes undefined. I think this might have to due with it not passing into the function?

Update 3

My
undefined
issue was caused by a missing semicolon, ending the for loop early. I have rectified it, but now I have a Max Stack Trace issue again.

My question context seems to now be "How do I make this
for
not run async?"

Answer Source

Here is your problem: for (var k=0;k=brands.length;k++){ You have used the operator = which is the assignment operator and put the length of brands into the variable k instead of the < operator that check if k is still smaller then brands.length.

What happen is that the expression tested in the loop is the number 8 (the length of brands array) and it is always remains 8 so you're getting into an infinite loop.

Notice that with boolean expression any value different than 0 represent the value true while 0 represent false.

This is about the infinite loop problem, also notice there is another problem with your code with passing the token to dataRequest function.

The problem is your trying to pass it immediately after the req.end(); which send the request.

At this point the response still not there and the callback function of res.on("end", ... is still not executed. basically you are passing undefined to the dataRequest function which cause your second error. simply move the call to dataRequest to happen inside the end callback like so:

res.on("end", function () {
   var body = Buffer.concat(chunks);
   var json = JSON.parse(body);
   key = json['access_token'];
   dataRequest(key);
});
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download