Mathieu M-Gosselin Mathieu M-Gosselin -4 years ago 43
Javascript Question

Events system causing stack overflow in Javascript

I will try to explain my actual setup, the idea behind it, what breaks, what I've tried around it.

The context



I have a PHP5.3 backend feeding "events" (an event being a standard array containing some data, among which a unique sequential number) to Javascript (with jQuery 1.7.x). The events are retrieved using jsonp (on a subdomain) and long-polling on the server side. The first event has the id 1, and then it increments with each new event. The client keeps track of the "last retrieved event id", and that value starts at 0. With each long-polling request, it provides that id so the backend only returns events that occurred after that one.

Events are processed in the following manner: Upon being received (through the jsonp callback), they are stored in an eventQueue variable and "the last retrieved event id" is updated to the one of the last event received and stored in the queue. Then a function is called that processes the next queued event. That function checks whether an event is already being processed (through the means of another variable that is set whenever an event is starting to get processed), if there is it does nothing, so the callstack brings us back to the jsonp callback where a new long-polling request is emitted. (That will repeat the process of queueing new events while the others are processed) However, if there is no event currently being processed, it verifies if there are events left in the queue, and if so it processes the first one (the one with the lowest id). "Processing an event" can be various tasks pertinent to my application, but not to the problem I have or to the context. For example, updating a variable, a message on the page, etc. Once an event is deemed "done being processed" (some events make an ajax call to get or send data, in which case this happens in their success ajax callback), a call to a another function called eventComplete is made. That function deletes the processed event from the event queue, makes sure the variable that handles whether an event is being processed is set to false, and then calls the function that processes the event queue. (So it processes the next, lowest id, event)

The problem



This works really well, on all tested major browsers too. (Tested on Internet Explorer 8 and 9, Chrome, Opera, Firefox) It also is very snappy due to the utilization of long polling. It's also really nice to get all the "history" (most events generate textual data that gets appended in a sort of console in the page) of what has happened and be in the exact same state of the application, even after reloading the page. However, this also becomes problematic when the number of events gets high. Based on estimates, I would need to be able handle as many as 30,000 events. In my tests, even at 7,000 events things start to go awry. Internet Explorer 8 stack overflows around 400 events. Chrome doesn't load all events, but gets close (and breaks, not always at the same point however, unlike IE8). IE9 and FF handle everything well, and hang 2-3 seconds while all events are processed, which is tolerable. I'm thinking however that it might just be a matter of some more events before they break as well. Am I being just too demanding of current web browsers, or is there something I got wrong? Is there a way around that? Is my whole model just wrong?

Possible solutions



I fiddled around with some ideas, none of which really worked. I tried forcing the backend to not output more than 200 events at a time and adding the new poll request after all the current queue was done processing. Still got a stack overflow. I also tried deleting the
eventQueue
object after it's done processing (even though it is empty then) and recreating it, in the hope that maybe it would free some underlying memory or something. I'm short on ideas, so any idea, pointer or general advice would be really appreciated.

Edit:

I had an enlightenment! I think I know exactly why all of this is happening (but I'm still unsure on how to approach it and fix it), I will provide some basic code excerpts too.

var eventQueue = new Object();
var processingEvent = false;
var lastRetrievedEventId = 0;
var currentEventId = 0;

function sendPoll() {
// Standard jsonp request (to a intentionally slow backend, i.e. long-polling),
// callback set to pollCallback(). Provide currentEventId to the server to only get
// the events starting from that point.
}

function pollCallback( data ) {
// Make sure the data isn't empty, this happens if the jsonp request
// expires (30s in my case) and it didn't get any new data.
if( !jQuery.isEmptyObject( data ) )
{
// Add each new event to the event queue.
$.each(data.events, function() {
eventQueue[ this.id ] = this;
lastRetrievedEventId = this.id; // Since we just put the event in the queue, we know it is officially the last one "retrieved".
});

// Process the next event, we know there has to be at least one in queue!
processNextEvent();
}

// Go look for new events!
sendPoll();
}

function processNextEvent() {
// Do not process events if they are currently being processed, that would happen
// when an event contains an asynchronous function, like an AJAX call.
if( !processingEvent )
{
var nextEventId = currentEventId + 1;

// Before accessing it directly, make sure the "next event" is in the queue.
if( Object.prototype.hasOwnProperty.call(eventQueue, nextEventId) )
{
processingEvent = true;
processEvent( eventQueue[ nextEventId ] );
}
}
}

function processEvent( event ) {
// Do different actions based on the event type.
switch( event.eventType ) {
case SOME_TYPE:
// Do stuff pertaining to SOME_TYPE.
eventComplete( event );
break;
case SOME_OTHER_TYPE:
// Do stuff pertaining to SOME_OTHER_TYPE.
eventComplete( event );
break;

// Etc. Many more cases here. If there is an AJAX call,
// the eventComplete( event ) is placed in the success: callback
// of that AJAX call, I do not want events to be processed in the wrong order.
}
}

function eventComplete( event ) {
// The event has completed, time to process the event after it.
currentEventId = event.id; // Since it was fully processed, it is now the most current event.
delete eventQueue[ event.id ]; // It was fully processed, we don't need it anymore.
processingEvent = false;
processNextEvent(); // Process the next event in queue. Most likely the source of all my woes.
}

function myApplicationIsReady() {
// The DOM is fully loaded, my application has initiated all its data and variables,
// start the long polling.
sendPoll();
}

$(function() {
// Initializing my application.
myApplicationIsReady();
});


After looking at things, I understood why the callstack gets full with many events. For example (-> meaning calls):

myApplicationIsReady() -> sendPoll()


And then when getting the data:

pollCallback() -> [ processNextEvent() -> processEvent() -> eventComplete() -> processNextEvent() ]


The part in brackets is the one that loops and causes the callstack overflow. It doesn't happen with a low amount of events because then it does this:

pollCallback() -> processNextEvent() -> processEvent() -> eventComplete() -> sendPoll()


That would be with two events, and the first one containing an asynchronous call. (So it gets to the second event, which doesn't get processed because the first one isn't done processing, instead it calls the polling function, which then frees the whole callstack and eventually the callback from that will resume the activity)

Now it is not easy to fix and it was designed like that in the first place, because:


  • I do not want to lose events (As in, I want to make sure all events are processed).

  • I do not want to hang the browser (I can't use synchronous AJAX calls or an empty loop waiting for something to finish).

  • I absolutely want events to get processed in the right order.

  • I do not want for events to get stuck in the queue and the application not processing them anymore.



That is where I need help now! To do what I want it sounds like I need to use chaining, but that is exactly what is causing my callstack issues. Perhaps there is a better chaining structure that lets me do all that, without going infinitely deep in the callstack and I might have overlooked it. Thank you again in advance, I feel like I'm making progress!

Answer Source

How about instead of calling functions recursively, use setTimeout(func, 0)?

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download