We're currently working on a way to dump all threads from the current app pool somewhere when an event occurs to narrow down the blocking issue. This is getting fun, processes dumping themselves.
We are definitely calling this method Dumpster.Fire();
After some great work from @m0sa this works - moving to enable this ability in prod:
@m0sa Just waiting on another occurrence now - we should get a minidump to find the jerk. Connection pool exhaustion we theorize is a red herring on async continuations failing to fire as a symptom of thread pool starvation, not the instigator of the event.
@m0sa It turns out the .NET 5 SDK is eating way more CPU while building than the 3.1 SDK did - we're trying to figure out why. It pegs all cores (github.com/dotnet/sdk/blo…), but perhaps Roslyn wasn't able to peg all cores before. @jaredpar any idea if there were threading changes in 5?

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Nick Craver

Nick Craver Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @Nick_Craver

17 Dec
The last 3 days is one of the longest deep dive debugs we've had in years. We finally got the culprit for some timeouts this afternoon: the compiler. It eats more CPU in a later version, not by much, but enough. Here's a segment of our build with .NET SDK 3.1.402 vs. .NET 5.0.101 ImageImage
Note: we've run builds on our web tier for the past decade. Why? Because they're typically at 5-10% CPU and we don't have a lot of hardware - so we used the spare capacity. Now, with Roslyn pegging all cores in that spike, it finally hurt.

Let's walk through the past few days.
Typically, our web tier is very idle, you could say it's boring. Thread pool queues sit at 0, the largest app sits at 1-4% CPU, and concurrent requests are generally sub-10 (because they're in and out fast). It looks like this (last 7 days - note the spikes the past few days): ImageImage
Read 14 tweets
16 Dec
I explained this a few times today, and I think it's important for every dev to know: timeouts are not necessarily what they seem.

A timeout happens when an operation doesn't complete fast enough. But, what does that mean? What are the details of that check? Well...
You almost never check timeouts live, unless doing so is VERY critical and worth being a primary CPU consumer. That's very expensive. Usually, what happens is one of 2 things:
- A timeout event is queued, and if it goes off first: boom.
or:
- Timeouts are checked on an interval.
If you check timeouts "live", well...nothing's "live". It's some period of time, even if it's 1ms.

To queue a timeout with the event and you're checking both: that means another item to track and schedule, which means cost, that scales.

On an interval, you sacrifice fidelity.
Read 8 tweets
23 Oct
I've seen the question "what is the best gaming WiFi" several times this week.

The answer is "cat5e or higher".
Yes that's a smartass answer, but seriously: if you can at all possible: cable!

WiFi is, in most homes, a time-share based model and the weakest/lowest device still dictates some of the overall behavior. It's not the bandwidth, it's the stalls when it's other people's turn.
I've covered this before, but it's worth saying again: we represent units in seconds because that's how *humans* can reasonably compare them. Computers operate in nanoseconds. You need to break down what's inside a second, then the lower time intervals matter for your purpose.
Read 7 tweets
22 Oct
I am really not a fan of GitHub Actions change. A YAML file format for key value pairs and...we can't use it? Environmental variables for a build are super common, at least for me, and now all config support is tossed away:
github.blog/changelog/2020…
Okay my fault! If you're hitting this it is *not* the env: lines (I'm going to poke that this get clarified in the docs, since the only way listed is the other way). env: works fine, it's an older version of Nerdbank.GitVersioning (not that old) causing it here...upgrade to fix.
Props to @m0sa who's already hit this for pointing me at the cause...I would have hunted for a while there.
Read 4 tweets
8 Oct
I cannot wait to *not* maintain a build server anymore.
Building on the latest software on agents you maintain is fine when stable, and not fine when moving fast on platforms.

To canary Stack Overflow on .NET 5 I need to build on TeamCity, but it's not detecting .NET 5 right, so I need to upgrade, which means chasing a license and...
Yeah, it's yaks. I just want to run dotnet build and you think that'd be simple but no - it can't be simple. They have to do fancy detection and lots of mess.

On GitHub Actions we'll be able to just pick an SDK version to install and go anytime - a self-contained app to deploy.
Read 4 tweets
8 Oct
Was using the MacBook on my bike just fine about 20 minutes ago…opened the lid and no display (reboot, etc. doesn’t fix…trying more resets). How’s your morning going?
I’ve tried all the keyboard-based resets: no dice. Time to take it apart and yank the battery.
“The new normal”
Read 11 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!