Production is very, very rarely debugged in a debugger.
This warrants some explanation. Let me elaborate. When debugging something, you generally use a few approaches:
- Attach a debugger
- Artifacts from an event (e.g. an error log)
- A quick high-level debug dump of a running process (e.g. stack trace)
- Full memory dump
- Guessing
- Attach a debugger
- Artifacts from an event (e.g. an error log)
- A quick high-level debug dump of a running process (e.g. stack trace)
- Full memory dump
- Guessing
When you attach a debugger, you have to:
- Have said debugger installed
- Setup production around a debugger to some extent, e.g. GUI on the running machine
- Be willing to stall the process (e.g. timing out requests)
While 1 and 2 may be doable, number 3 is a hard no at scale.
- Have said debugger installed
- Setup production around a debugger to some extent, e.g. GUI on the running machine
- Be willing to stall the process (e.g. timing out requests)
While 1 and 2 may be doable, number 3 is a hard no at scale.
Artifacts after an event usually includes:
- Traffic logs
- Error logs
- Eye witness descriptions of the suspect
- Terrible bug reports
- Build logs (what was deployed?)
- Source control logs (what was the change? the reason?)
- Metrics/monitoring system logs
- Traffic logs
- Error logs
- Eye witness descriptions of the suspect
- Terrible bug reports
- Build logs (what was deployed?)
- Source control logs (what was the change? the reason?)
- Metrics/monitoring system logs
A high level dump can give you overall memory usage, stack traces running (e.g. touch/time profiling summation), or possibly some other light-touch non-interruptive data collection.
A full memory dump is very handy *for a point in time and only a point in time**, but is very expensive. The amount of time an application will stall (and number of requests dropped) will scale with the amount of memory in use. Shadow usage from page pins will also cause a rise.
This shadow copy of pages in delta during the memory dump there. That's basically how some dumps work. Then depend on the setup, you may have to move the memory dump elsewhere to analyze it. This all takes resources at scale...so it's rarely a default approach.
So anyway, and this is just my experience, artifacts you can collect at the time of the event are paramount. They're your bread and butter as long as you're not allergic to bread. That's where you start and hopefully end. It's info easy to aggregate and analyze.
When approaches from frameworks and platforms focus on the debugger experience, I consider this a local luxury only. The debugging experience is far greater...and often doesn't include a debugger at all. If you're designing for these scenarios, please keep production in mind.
It looks like we'll be contributing to this space for .NET Core in the artifacts area as well as process light-touch dumping. For example getting stack traces without killing the process. Top level memory data, etc. We're going to work with the .NET team and see what's possible.
What I love is when we're talking about this, I was pointed to @anurse's work already well underway here: github.com/aspnet/AspLabs… - this reflects some of the stuff we have internally (but can't open source) today that we need to make work on Core. Awesome! We'll collaborate.
