,
6 tweets,
2 min read
Read on Twitter
A good reminder from @lizthegrey in her #QConLondon talk (paraphrasing) “If a single blade of grass is brown, it's not a problem. We're not going to replant the whole lawn for that. Same things for our services: it's not necessarily a problem if a single thing is down.”
Also expressed by @mipsytipsy as “nines don't matter if users aren't happy”: you can have 99.999% uptime, it's useless if users are not happy about the product. Conversely, it's OK to have “only” 99% uptime if it doesn't significantly affect user experience.
Also expressed by [can't remember who?] as “in any sufficiently complicated system, there is always at least one component broken somewhere”. 100% uptime doesn't exist in complex systems!
Consequence: it's important to have meaningful Service Level Indicators and Service Level Objectives. (Uptime is rarely a meaningful SLI.)
Another consequence: alerting should be done on the SLI, not raw metrics.
And preferably on *trends*; i.e. not at the moment when the SLI crosses the SLO threshold, but when a predictor anticipates that it *will* cross soon.
And preferably on *trends*; i.e. not at the moment when the SLI crosses the SLO threshold, but when a predictor anticipates that it *will* cross soon.
This means that many alerts then don't need to be emergencies: if we know that we will run out of disk within 5 days, we don't need to page someone in the middle of the night. It can be deal with during normal office hours.
💯
💯
