CloudWatch Anomaly Detection is of the devil, but only if you use it wrong. A thread:
So CloudWatch Anomaly Detection launched today, and is generally awesome. See @jeffbarr's blog post at aws.amazon.com/blogs/aws/new-… for a great overview.
For the impatient, this feature is basically using Exponential Smoothing (en.wikipedia.org/wiki/Exponenti…) buried under the umbrella of Machine Learning to develop confidence intervals around CloudWatch metrics.
"If something spikes sharply and out of a confidence interval, it fires an alarm" is a great signal for someone to investigate further. It's freaking lousy as far as reasons to wake someone up at 2AM go.
Maybe there's a monthly backup that wasn't observed. Maybe a blog post went viral. Maybe us-east-1 just exploded. Not all of those justify waking up a human.
This is a great feature; don't get me wrong. Just don't use it as your sole source of signal to awaken an angry SRE; their postmortems may be blameless but they will absolutely blame you for waking them up.
So what I'd do is set up a trial model. See if it works for you. Have the alarm go to a Slack team, or somewhere folks will see it during business hours during an adjustment period where you see if it works for you and your environment.
And for god's sake turn it off before you do something abnormal like a Superbowl ad.
I'm adamant on this point largely because I had a job once with such a crappy on-call environment that "waking someone up" is now reserved for "the world is ending" level events.
So in conclusion, good feature, great pricing (free except for the usual cost of CloudWatch alarms), and it's super low-effort to test out in your environment.
Just please let me sleep in a bit longer when us-east-1 melts.
