Thread by @mipsytipsy

I wrote a post about the emerging generational gulf in observability tools.

O11y 1.0: many pillars, many tools, lots (but not all) are metrics backed. APM, RUM, tracing, logs, etc

O11y 2.0: wide, structured canonical log events are your source of truth

charity.wtf/2024/08/07/is-…

It's never been simpler to understand the generational divide -- no laundry list of features to memorize and reason about -- and never been easier to understand why it matters so much.

Either your time and attention are scattered across many irreconcilable data sources, or not.

The technical distinction between 1.0 and 2.0 is so minor that it's easy to underestimate the seismic importance, and the waves of sociotechnical transformation this unlocks.

There are two historical analogues I can't stop thinking about: virtualization and TDD.

Bear with me...

Virtualization is old, old tech. We've had VMs since the 70s! But it wasn't til VMware productized it in 1999 that shit got interesting. That opened the door to cloud computing and mainstream SaaS.

It's also what sparked 💥the DevOps movement.

Before VMs, ops teams were bound to hardware and data centers, operating systems and deps. Software engineers lived in application land, and the overlap between the two worlds was *miniscule*.

That all began to change once you could write code to manage your infrastructure.

It took us like 15 years to get from VM-as-a-product to, like, "stop hugging servers" and "cattle not pets". A decade or more of people figuring out better workflows, building better tools, hooking up feedback loops that lets engineers move more and more swiftly, with confidence.

These days, I feel like we are in the waning days of devops. (Which is not the same as saying "devops is dead". 🙄) Rather, the problems devops was invented to solve have increasingly *been solved*.

We don't have siloed ops and dev teams who need to learn to work together.

Increasingly, we live in a post devops universe, where every engineer writes code, and every engineer owns the code they wrote. As God intended. 😉

This is a good thing!! Of course, now we have new problems. One of them is that our o11y tools are a relic of a siloed age,

...where front end, back end, and ops were all expected to look at different data and care about different things. And at the time hardware was insanely expensive, so we threw away alllll the context and kept just the summary metrics.

(Oooof.) Anyway, that's how we got o11y 1.0.

The other historical moment on my mind right now is also circa 1999, when @KentBeck (re)discovered TDD, test-driven development.

Up til that point, people basically just wrote code and crossed their fingers, waiting for users to tell us if anything broke or regressed, lol.

@KentBeck TDD is when programming started to become software engineering, imo. Not because you necessarily have to do TDD per se, but you do need to have *tests*. TDD made it clear to every developer that YOUR code is YOUR responsibility.

*You* own the quality of your code, not QA.

@KentBeck It's hard to overstate what a revolutionary concept that was. Or how utterly necessary it has turned out to be.

However, the effectiveness of TDD and tests in general have been losing ground steadily over the past ten or fifteen years.

I'm not saying, "don't write tests". You should still write tests! It's the fastest, cheapest way to make sure you didn't do any seriously dumb shit.

But the fact that your tests pass doesn't mean your code works. It means your code is probably logically correct, and that's all.

Back then, you were probably working on a monolith application in a simple architecture -- web, app, data. If your tests passed, and you had good tests, you could feel reasonably confident in it. The overwhelming majority of the complexity was bound up neatly in the app tier.

Now your code is running...where? Maybe one of your hundreds or thousands of microservices, maybe storage service or proxy, maybe PaaS, SaaS, serverless, lambda jobs, client side compute or cache on your phone or web or IoT device ... who the fuck knows?

And how is it going to perform? Lol if you think you can write a test to simulate the concurrency, emergent behavior, thundering herds, or flighty user whims of the day.

You can't. No one can. Staging is just a glorified laptop. Only prod is prod.

Twenty-five years ago, engineers began to realize that the only way to ship consistently good code was for devs to embrace tests.

That's how feedback loops work. You can't say "ok, THESE people are going to write the code and THOSE people are going to make sure it works" 👍😬

And now, in the waning days of the devops era, I think we're seeing a similar seismic shift around operability and production systems.

The only way to ship consistently good code is to embrace a development workflow that includes instrumentation, observability and prod.

You can't say "ok, THESE people are going to write the code and THOSE people are going to understand the code." Understanding is part of writing.

I think of this as ODD, or "observability driven development" (but @jessitron prefers "observability during development" 😉).

ODD means: you instrument your code as you go, asking yourself "how will I know if this is working or not?"

Then you deploy it. Then you inspect your code through the lens of the instrumentation you just wrote.

1, is it doing what you expected?
2, does anything else look weird?

Your job isn't done when you merged your code and tests pass. Your job is done once you've verified it in production, and maybe let it bake for a bit.

Or at least, that's the bare minimum. Here's where we finally circle back to the original topic, observability 1.0 vs 2.0.

Testing before production is great, and you should do it. But you run into diminishing returns really fucking fast, because it's all fake and controlled.

If you care a lot about the quality or stability of your code, or the user experience, you *have* to invest in production.

There's been an explosion of innovation in this space, and this is just the beginning.

- progressive deployments
- canaries
- feature flags

However, most companies are still using o11y 1.0 tools and this is ... more of a problem than you might think.

The beauty and joy of production tooling like this is all about taking a scalpel to production traffic.

"What happens when I flip this flag?"
"What happens when I promote this canary from 1% to 5%?"

But with metrics-backed tools, you only get averages and randoms. No scalpels.

By averages and randoms, I mean: you can get buckets (mean, median, 90%, 99%, 99.99%), and you can get random exemplars.

What you can NOT do is ask precise questions to go with your precise tooling. With o11y 2.0, you could see _exactly_ what happened with your flag or canary --