NVO NVO - 1 year ago 121
Node.js Question

NodeJS async large dataset callback

I have an problem with processing a huge dataset in NodeJS. I'm using async because I want to call a

when everything is complete.


In a MongoDB collection on this project it is possible to add campaigns. Shops are able to join those campaigns and add some actions from that campaign to there agenda. (like post Facebook message, of photo to Instagram)
On the frontend of the site, there is a page which shows all active campaigns. When a campaign is opened, there must be an list with all the shops which joined this campaign.

Because every shop has an agenda, where is saved which tasks are to do, I must loop through all shops to see if there is an action with a campaignId what is equal to the ID from the campaign what is open.
There are more than 3500 entries in the shop collection, so it can take a while when every shops is processed, so I decided to use angular to post that request when the page is loaded.


As you can see in the code below, I get all the shops, and loop through the items. Then I loop through the agenda items to see if there is an matching campaignID, then I add some info in an array what I want to send back. But the code stops after 1 shop. How can I add all the shops in the array, and send all when all shops are processed?


(not all fields)

var Shop = new Schema({
phone: String,
campaignName: String,
downloaded: Boolean,
success: Boolean,

Code in controller:

module.exports.getCampaignShop = function(req,res){
var campaignId = req.query['campaignId'];
console.log("campaignId", campaignId);
var result = [];
Shop.find().sort({name:1}).exec(function(err, shops){
console.log("shop count", shops.length);
async.eachSeries(shops, function(shop, allDone){
async.eachSeries(shop.agenda, function(agenda, shopComplete){
if(agenda.campaignId == campaignId){
var shopResult = {
shopId: shop._id,
nameSlug: shop.nameSlug,
logo: shop.logo,
name: shop.name
console.log("shopResult", shopResult);
}, function(err){
if (err) throw err;
console.log("result length", result.length);


Answer Source

The issue is shopComplete callback get called only when agenda.campaignId == campaignId is True. You need to call it outside the if block.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download