Once someone has experienced how much easier it is writing code with real observability, you cannot pull it out of their cold dead hands. It's like getting glasses for the first time, and realizing you could barely see the world around you.
This is partly a generational thing. Those of us who grew up writing software with metrics and logs have a lot of unlearning to do, a lot of trauma and frustration to unwind.
Engineers who never learned to navigate monitoring tools actually have it much easier picking up o11y.
It really is hard to explain. You kind of have to experience it yourself. This is why we have such a generous free tier, to encourage engineers to play around and do real things with it.
When it clicks, it clicks. You'll see what I mean.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
I would actually argue that, with the right tooling, production is *exactly* the most effective place to take chances, make mistakes, and get messy. 🙃 Otherwise you're like this guy.
Here are the slides for a talk of Liz's that I modified slightly and delivered on Wednesday for the O'Reilly InfraOps superstream. speakerdeck.com/charity/observ…
We walk you through the honeycomb backend, some of the ways we perform chaos engineering, and some infamous outages,
to show just how swiftly, accurately, and powerfully you can manipulate systems with modern tooling (feature flags, fast delivery, superb observability) and do whatever the fuck you want in prod without hurting your users.
LARGE SYSTEMS USUALLY OPERATE IN FAILURE MODE, via @dangolant
Or like I used to say, your distributed system exists in a continuous state of partial degradation. There are bugs and flakes and failures all the way down, and hardly any of them ever matter. Until they do.
This is why observability matters. SLOs make large multitenant systems tractable from the top down, but observability makes them comprehensible from the bottom up.
Maybe only .001% of all software system behaviors and bugs ever need to be closely inspected and understood, but that tiny percentage defines the success of your business and the happiness of your users.
And you CANNOT predict what will matter in advance.
bodies have limits
bodies have limits
bodies have limits
bodies have limits
bodies have limits
bodies have limits
bodies have limits
bodies have limits
bodies have limits
bodies have limits
bodies have limits
bodies have limits
bodies have limits
bodies have limits
bodies have l
I was homeschooled, and escaped to college when I was 15. I was a seething mess of pent-up rage and ambition (and undiagnosed ADHD) who had never done any sort of formal schooling. I had no idea what I wanted to do other than ALL OF IT. RIGHT NOW.
You're supposed to register for 12-15 credits, so I promptly registered for 24 (plus I had a piano performance scholarship I was supposed to maintain).
I didn't have any family support, money, or ability to take out loans, so I signed up for three local minimum wage jobs.
it's a bit counterintuitive, but the better-instrumented and the more mature your systems are, the fewer problems you'll find with automated alerting and the more you'll have to find by sifting around in production by hand.
hey man, you know me, I don't like talking smack about others, and I'm not sitting over here whittling and looking for excuses to litigate people's usage of the word observability.
they are literally describing monitoring. good ol', 30-year-old traditional monitoring.
* Notify
* Triage
* Understand
this is a company with a billion dollar valuation and they literally don't know the difference between monitoring and observability
i mean, we can all argue over the subtleties of observability and that's relatively understandable, but doesn't fucking EVERYBODY know what *monitoring* is and does?
good morning kittens, guess what honeycomb been up to? ? oh not much really, we've only just STAVED OFF OUR OWN INEVITABLE DEMISE AND DESTRUCTION, 🔥YET AGAIN🔥.
We can hardly even fail if we try for another two, three years now! Take that, heat death of the universe!🪐🌑 💜
I wonder if it will ever stop feeling so bizarre just to still exist. 🙃 The list of people we are grateful for and permanently indebted to gets longer and and longer and longer with each passing year.
From our investors, who are principled, curious, endlessly thoughtful and helpful -- nothing like the stories and stereotypes about VCs that tend to filter down to eng circles -- to our family members, especially anyone who had to live with us those early few years 😬