I'm hosting an ASP.NET web API, which has been in production for several years and which has seen literally dozens of updates. All of a sudden, after only minor changes in the code, the API crashes after exactly 50 requests, no matter which browser or client is being used. A web server restart immediately restores API availability (application pool recycling doesn't), but the mysterious countdown begins anew...
This weird effect persists across several machines (Windows Server 2008, Windows Server 2013, and Windows 7) and across different web servers (IIS 7.5, IIS 8.5, UltiDev Web Server). IDE is Visual Studio Community 2015; .NET version is 4.5.2.
I haven't added foreign API dependencies. I'm suspecting Firebird (the only external dependency) of imposing that limit of "50," but all my experimenting and googling was to no avail.
The only halfway useful log message I could extract said that the DB connection failed due to a timeout error. (I'm using Firebird 2.5 SuperServer x64 and Firebird .NET Provider with a FirebirdSql.Data.FirebirdClient.dll of version 3.0.2, which has served me well so far.) Still, I can connect to the DB and browse it via FlameRobin. In any case, everything works fine as soon as I go back to the previous API version.
After the crash, a previously cleared firebird.log contains 50 identical messages like this:
AB-01234-CD (Server) Fri Nov 25 14:22:57 2016
INET/inet_error: read errno = 10054
That's what it was. The mysterious "50" is the maximum pool size in the connection string. The bug, which I was chasing for days, is one of simplest I've ever seen -- the line where the DB connection is done was duplicated. Like this:
Since only the second connection got closed, every request sent one connection into limbo, and consequently the connection pool was quickly crowded. This happens if you chance a Friday night release!
Deleting one of the two identical lines solved the issue, of course...