Thread by @Nick_Craver

We're trying to repro an error we can only catch in production under extreme load (after many hours of hunting). I'm about to re-enable the problematic server now to get a dump.

You few hundred users who are about to get 500s, we salute you for your sacrifice.

Update: we haven't found the *exact* issue but we found super curious details which tremendously narrowed it down (exact route) and a mitigation. We'll try to repro and eliminate the cause, but have a workaround for anyone on-call now as well.

Your sacrifice is appreciated!

If you're curious, we have this rare (once every ~ 2 weeks) issue where our connection pool (150 max) fills up and *never* releases. We can take the server out for an hour, it'll come back still full. The 150 never let go. The connections are lost in the ether for some reason.

We knew we didn't have data last time, so we set up an alert to ping us in chat when the thread pool exploded on any web server for that pool.

This morning, we caught it quickly and got a memory dump. This dump has connection tracking data we added a while back, it has...

Connection details that we track for each pool, on every SQL connection Stack Overflow (and all sites) make. We see what lingers open, what the last command run was, what the flags were, what the ISO level is, etc.

This let's us dig in and see what the "state of the world" is.

Next up, @marcgravell is going to add a Redis pub/sub-backed route which takes this data, protobuf encodes it, and shoves it in a Redis hash (keyed per server). This will let us get a global overview from any server (since accessing the problematic server is...problematic).

@marcgravell Anyway, one of the pieces of information (hard to see in the tooltip) is the #AspNet MVC route name. So we can see if we have tons of open connections last running a command from route X, our problem/leak is almost certainly in that route path. In this case, our Jobs proxy.

@marcgravell We also have this fancy redis pub/sub-backed buttons/routes that clear connection pools across all our servers (non-disruptively). We built this for SQL AG changes (e.g. read-only routing), but it works as a mitigation here. It doesn't lower the thread pool, though.

@marcgravell I was able to capture the tooltip from earlier a bit better - here's a screenshot: