I know that this is a known and discussed problem, but I would just like to get the dimensions right here:
I am running
Ubuntu Sever 16.04 node (12 cores, 256G ram)
ulimit to > 130k
Here is what turned out to be the case:
We switched to Jest Client which significantly reduced the load on both client and server. 900 concurrently active clients now result in <2000 file descriptors on the server.
Thanks to Andrei Stefan for pointing us into the right direction.