Elad Nava Elad Nava - 5 months ago 66
Node.js Question

Node.js Elastic Beanstalk refuses connections under heavy load (ECONNRESET)

I deployed a simple Node.js Express app to Elastic Beanstalk.

Using the loadtest npm package, I hit up the EC2 instance directly (bypassing the ELB) with 150 requests per second, and that's when things started getting hairy.

loadtest -c 10 --rps 150 http://{EC2-IP}/


The server would reset the connection immediately (
ECONNRESET
), rending the app unreachable. It refused to support more than X concurrent connections.

Answer

Upon diving into the EC2 server's logs, I found the following log message spammed across the file:

1024 worker_connections are not enough

I then understood that I need to tweak the worker_connections declaration in /etc/nginx/nginx.conf, as well as the increase OS file descriptor limit, since each connection requires 1 file descriptor.

You can check the OS file descriptor limit by running:

ulimit -n

However, since the instance is managed by Elastic Beanstalk, Amazon asks that we instead make our edits to /tmp/deployment/config/#etc#nginx#nginx.conf, which contains the same content as /etc/nginx/nginx.conf, but Amazon knows to copy it over to /etc/nginx/nginx.conf when the deploy completes, as well as tell nginx to use the new configuration.

I created an .ebextensions file that increases the OS file descriptor limit, as well as the worker_connections declaration. It makes it possible to reach a theoretical limit of 1,000,000 concurrent connections, but obviously more factors are going to come into play before you reach that, such as the TCP stack configuration, CPU/RAM, etc. In any case, increasing this limit will allow us to support sudden spikes of traffic instead of returning ECONNRESET.

Include it with your application when you deploy, in the following path relative to the root directory of your deployed directory:

.ebextensions/nginx.config

files:
  "/etc/security/limits.conf":
    content: |
      *           soft    nofile          1000000
      *           hard    nofile          1000000
container_commands:
    01-worker-connections:
        command: "/bin/sed -i 's/worker_connections  1024/worker_connections  1000000/g' /tmp/deployment/config/#etc#nginx#nginx.conf"