Thread by @mipsytipsy

I would actually argue that, with the right tooling, production is *exactly* the most effective place to take chances, make mistakes, and get messy. 🙃 Otherwise you're like this guy.

https://twitter.com/robbiedelprete/status/1482061871628763136

Here are the slides for a talk of Liz's that I modified slightly and delivered on Wednesday for the O'Reilly InfraOps superstream. speakerdeck.com/charity/observ…

We walk you through the honeycomb backend, some of the ways we perform chaos engineering, and some infamous outages,

to show just how swiftly, accurately, and powerfully you can manipulate systems with modern tooling (feature flags, fast delivery, superb observability) and do whatever the fuck you want in prod without hurting your users.

I even included the full speakers' notes, since there isn't a ton of text on the slides. (You're welcome.)

It culminates in this ridiculous yet totally serious conversation, which I love.

The moral of the story is, error budgets are there to be used to find problems during peak daytime hours, so you don't find them in the middle of the night.

USE them.

My other favorite part of this talk is towards the very end, where we talk about the Kafka Month of Pain that didn't result in blowing any customer SLOs, but did exceed our internal human SLOs. Which are just as important.

Honeycomb on call teams have an SLO of not getting paged more than twice a week, or having to work an incident outside of business hours more than once every six months.

We were burning out our people, so we had to call a halt to the flurry of changes.

Also, I learned while preparing for this talk that we have an official policy that "incident responders are encouraged to expense meals for themselves and family during an incident", which just fucking makes sense once you think about it.

One of our core values is that "we hire adults", and adults have responsibilities outside of work. You cannot build a healthy, sustainable sociotechnical system if you don't account for that.