oh no, oh no no no no no 😱 "drowning in alerts and can't understand your systems? pay lots of money for an AIOpsBot you don't understand. to filter, route, and escalate them down to a manageable roar!" blog.newrelic.com/engineering/wh…
my sweet summer child, please do absolutely none of these things.
the best way to manage noise is to eliminate it. you are better off deleting 90% of your alerts (the ones that don't align directly with user pain), moving to SLO-based alerting based on real user pain.
the best way to manage noise is to eliminate it. you are better off deleting 90% of your alerts (the ones that don't align directly with user pain), moving to SLO-based alerting based on real user pain.
THE ONLY THING you can do to stay sane and on call is keep a tight lid on alerts and align them with user pain.
THE WORST THING you can do is accept defeat on this front, allowing alert spam to grow unchecked and unbounded, a fire hose that would kill any human that consumed it.
THE WORST THING you can do is accept defeat on this front, allowing alert spam to grow unchecked and unbounded, a fire hose that would kill any human that consumed it.
It may be harder in the short term to attempt to understand each alert and ruthlessly purge those that aren't well understood and tightly aligned with user pain.
I guarantee you it is compoundingly harder if you don't. You cannot outsource understanding your systems.
I guarantee you it is compoundingly harder if you don't. You cannot outsource understanding your systems.
This is tied up with the need to shift from monitoring to event-based observability, of course, so you have the right tools to debug your complex systems.
People have relied on their pager spam to triage and debug systems problems for too long. This shift can be challenging.
People have relied on their pager spam to triage and debug systems problems for too long. This shift can be challenging.
But paging alerts are not an appropriate tool for debugging your systems. Using clusters of paging alerts to diagnose and debug systems problems is like using a defibrillator to itch a mosquito bite.
Get you a better debugger. One that won't kill you.
Get you a better debugger. One that won't kill you.
