Wexoni Wexoni - 1 month ago 11
Node.js Question

Process and send large amount of data using Hapijs

We are using Hapijs/React/Sequelize/MySQL. There is something that consuses me, and I have made some research in to the matter.

Setup:

We have React component that issues AJAX call to Hapi API. API returns a array of objects. That is being displayed by React.
Issue:
We request data from a table. That data is returned as array of complex objects. Each of those objects is a sequlize object. And it has many additional properties that we do not need.
We send that object using HAPI reply, something like (40k objects):

flattenedParts.forEach(function(item) {
item.dataValues.Product = productsMap[item.dataValues.ProductId] || [];
item.dataValues.PartModule = partModulesMap[item.dataValues.PartModuleId] || [];
fullObjects.push(item);
});

reply(fullObjects).code(200);


When the code reaches reply statement, it takes 14 seconds until is send it as simple array of JSON objects.

For me it seems that serialization of those objects takes a long time. So I thought about making an array of plain objects, with only data that I need. I did something like this (40k objects)::

flattenedParts.forEach(function(item) {
item.Product = productsMap[item.dataValues.ProductId] || [];
item.PartModule = partModulesMap[item.dataValues.PartModuleId] || [];
var flat = new partCreator(item);
newParts.push(flat);
});

reply(fullObjects).code(200);


The part creator function is simple as this:

function partCreator(part){
this.Product = {};
this.id = part.dataValues.id;
this.PartNumber = part.dataValues.PartNumber;
...
}


So at the end I end up with array with simple objects containing the data that I need. The speed is now around 7 seconds. Almost 50% gain in speed.
But if I am correct it should be much faster.
What is going on behind the scenes with the reply function, and how can I speed this up?

Answer

There are not a lot of things to do to utilize the performance when you want to return back 40K(!) objects. You are going to have scaling issues really soon, on a large scale you can' t handle and process so many objects in memory on your server. The purpose of a REST API is to receive a request and to return back a small amount of data really fast. You have to limit the amount of data that you load into memory and should not be more than 100 (even 100 are a lot). You have to extend your API to support all the functionalities that you want (pagination, sorting, searching etc) and to extend the client to fire more requests to your API. By the way, some of the browsers that are running on devices that are not quite fast are going to have performance issues as well.

Comments