CoffeePeddlerIntern CoffeePeddlerIntern - 4 months ago 21
Node.js Question

Delaying promises: .delay() doesn't seem to work

This is an extended question from Throttling requests to 3rd part Apis with Node . You can see my full code in that post if you want to see the sub functions and a little backstory of what im trying to do.

What i need to do is delay api request to an api. I am hitting the server max per second. I am trying to use Promise.delay but it doesnt seem to hold anything. This is how im using it.

function getAllRegions(req, res){
getAllRegionsHrefs().then(function (hrefs){
var chunks = _.chunk(hrefs, 25);
return Promise.map(chunks, function(chunk) {
return Promise.map(chunk, getRegion).then(function (getRegionResults){
for(var item in getRegionResults) {
Promise.map(getRegionResults[item].constellations, getConstellationInfo).then(function (constellationInfo) {
var chunks = _.chunk(constellationInfo, 150);
return Promise.map(chunks, function (chunk) {
return Promise.map(chunk, getSystem).delay(20000);
})
}).delay(20000);
}
}).delay(200000);
});
});
}


Now this will make just about 9k api requests and it builds a universe based off regions, constellations and systems. Is there a better way of throttling the nested requests? At each nest, the number of calls exponentially goes up.

Update



Still doesnt seem to want to wait. Did I mess up the nesting again? I added 1 more function in there that was needed for the flow. Also the chunking seemed to wrap a double array so I took it out.

function getAllRegions(req, res) {
getAllRegionsHrefs().then(function (hrefs) {
return Promise.mapSeries(hrefs, function (href) {
getRegion(href).then(function (regions) {
return Promise.mapSeries(regions.constellations, function (constellation) {
getConstellationInfo(constellation).then(function (constellationInfo) {
return Promise.map(constellationInfo, getSystem).delay(15000);
});
});
});
});
});
}


Error (per request):



Unhandled rejection RequestError: Error: connect ETIMEDOUT

Answer

Promise.map starts all requests in parallel, so your code runs like:

  • requests -> delay
  • requests -> delay
  • requests -> delay
  • ....

But the desired behavior should be sequential:

  • requests -> delay -> requests -> delay -> requests -> delay ...

each chunk of requests should wait for delay of previous chunk.

You could achieve this by chaining promises:

startRequests(chunks[0])
.then(function() {
  return startRequests(chunks[1])
}).then(function() {
  return startRequests(chunks[2])
}) // ...

or use mapSeries

function getAllRegions(req, res){
  getAllRegionsHrefs().then(function(hrefs) {
    return Promise.mapSeries(hrefs, function(href) {
      getRegion(href).then(function(region) {
        var chunks = _.chunk(region.constellations, 150);
        return Promise.mapSeries(chunks, function (chunk) {
          return Promise.map(chunk, getSystem).delay(10000);
        });
      });
    });
  });
}

only innermost requests are chunked, others are started one by one.

Comments