Thread by @RaeKnowler

The next talk: @rmedranollamas, Google SRE, talking about Service Levels and Error Budgets. #DevOpsDays

First info: there’s a Google SRE book from O’Reilly now, available at g.co/srebook. #DevOpsDays

How good is your service? A Service Level Indicator is a quantitative measure of how good one attribute of the service is. This should be a property that users care about! #DevOpsDays @rmedranollamas

Example: able to log in within 300ms? Gather data, set up exporting it to your monitoring, and then pick a few metrics to carefully define indicators for. #DevOpsDays @rmedranollamas

Understand your metrics: averaging monitoring datapoints over time on a monitoring graph can hide big peaks! So: when you get an alert but your SLI looks okay, drill down. #devopsdays @rmedranollamas

Use histograms and percentiles in preference to averages. If you focus on having a good median performance, you’ll miss terrible values at the 95th percentile. #devopsdays @rmedranollamas

How good *should* a service be? That’s a product question and always requires tradeoffs. The cost of marginal improvements grows exponentially. Ask product and dev management; they know the necessary info on what’s really required/worthwhile. #DevOpsDays @rmedranollamas

Your targets should be ambitious but achievable. “Latency always under 250ms” => you’re actually requiring 100% availability! Better: “under 250ms 99.9% of the time.” #devopsdays @rmedranollamas

There is one service (didn’t catch the name) that’s so reliable that they take it down 15mins a month—just to be sure no one builds on it expecting 100% availability, in case they ever have an unexpected outage! #DevOpsDays @rmedranollamas

Service Level Objective (SLO): what you publish to users, e.g. five nines uptime (99.999%). Aim to be a bit better than your SLO, but not too much better: users will come to rely on it. Oh, and do chaos engineering, but when it’s convenient, not at random! #devopsdays

An SLO implies an acceptable level of errors... an error budget. You can use this budget to accommodate rolling out new versions, unpredictable failures, etc. Watch the budget over the last 30 days and plan updates when you can afford some downtime. #DevOpsDays

Use alerts to keep track of your error budget exhaustion, e.g. if you’ve used 1% of the month’s budget in an hour. And keep a ‘rainy day fund’ for when you need to deploy a fix in an emergency. #DevOpsDays @rmedranollamas

SREs should work with devs to improve reliability *and* velocity. Build consistent systems for deployment, use load balancing and rollouts to reduce how much damage a bad update can do, create safe ways to rollback updates. #devopsdays @rmedranollamas

SREs also need the authority to stop releases if the error budget can’t accommodate them! #devopsdays @rmedranollamas

Finally, @rmedranollamas recommends doing the exercise to define an SLI even if you don’t plan to use it in production. :) #DevOpsDays

Q: do you also have SLOs about data quality, e.g. the same data returned in the USA, Asia and Europe?

A: yes—not Ramón’s own team, as authentication is quite binary, but other Google SRE teams do. #DevOpsDays

@threadreaderapp Please unroll.