Thread by @jessitron

SRE teams try to keep toil under 50%.

Only 50% of work that has no enduring value...

@DivineOps #QConSF

SREs in the audience? (Dozens of hands)
Experienced SREs? (Like 2.5 hands)

We @RedHat used to ship products. Build a thing, package it, send to customers. Then it was their problem. Customer hires a consultant or figures it out.

Now we mostly ship services. Now it’s our headache, reliability and uptime etc. It’s different

@DivineOps #QConSF

SRE: “hopefully people get paid more for having this title”

The innovative part of SRE is: explicit agreements that align incentives. Between dev, ops, business.

@DivineOps #QConSF

SLA = financially-backed availability.
The contract has a % of cost that is refunded if availability is lower than advertised.

This aligns incentives between vendor and customer.
(So much as a single metric can)

@DivineOps #QConSF

SLO = targeted reliability. What do we care about?
(Service level objective)

Example: availability from a customer’s perspective

@DivineOps #QConSF

SLI = actual reliability.
Without good monitoring, you don’t know whether the service does what the user expects it to do.

Monitoring improves as internal users catch stuff and move it into automation (only problems seen before)

@DivineOps #QConSF

SLO = business approved reliability.

Explicitly aligns incentives between Business & Engineering.

@DivineOps #QConSF

Error budget = acceptable level or Unreliability

When it’s gone, developers shift focus from delivering features to improving reliability.
… which they never had the incentive to do before!

@DivineOps #QConSF

100% availability… no. Impossible, unnecessary, extremely expensive.
99.999% … will your users even notice?
The background error rate of the internet is .01-1% (depends on ISP)

@DivineOps #QConSF

Things we got wrong about SRE:

the book says “it’s what you get when a software engineer designs an operations team” … no.

This led to:
Hire developers to do ops things, and get effective SRE? …no.

@DivineOps #QConSF

Why didn’t Ops automate themselves out of a job?
because they didn’t have a software engineer? no.

They didn’t have APIs! The only way to update a registry was a human clicking.

@DivineOps #QConSF

Kudos to Jeffrey Snover, who fought for PowerShell automation for Windows administration.

Google explicitly built Borg to be automatable.

Then people got the message: infrastructure as code.
Puppet, chef, etc.

@DivineOps #QConSF

As an industry, we worked really hard to make the tools to make this automation happen.
THAT is what makes SRE possible.

Consistent APIs and reliable monitoring are prerequisites to automation.

@DivineOps #QConSF

Second thing we got wrong:

Toil is bad, it’s useless, eliminate it.

Are we striving for a human-less system?

@DivineOps #QConSF

Humanless systems don’t maintain themselves. No matter how automated, how well structured — they don’t maintain their structure.

@DivineOps #QConSF

People do troubleshooting, responding, adapting, noticing
to keep the system functioning: it looks a lot like toil.
Are people rewarded for that?

Some SREs do more automation, others more on-call keeping the system running and learning from it.

@DivineOps #QConSF

Cloud provides an industry standard for consistent infrastructure-level APIs
❤️☁️❤️

Also kubernetes 🤩

@DivineOps #QConSF

Align your toil where your business value is.

Below that, call a PaaS API.

@DivineOps