Thread by @mipsytipsy

"I'd love to bring in honeycomb and observability, but my team isn't there yet, they're just learning basic monitoring. Maybe someday, after they've mastered that.."

I hear this a lot. Surprisingly, it's almost exactly backwards. THE WAY YOU ARE DOING IT NOW IS THE HARD WAY.

I'm not being contrarian or exaggerating. We are so used to monitoring tools, we forget how excruciatingly indirect they are.

Want to know if the fix you just shipped *actually* dropped the latency for EU users exporting large datasets after setting optional flags? Nontrivial.

Oops I need drinks brb

Picking this up a day later, but I have also had drinks tonight, so it's cool

First of all, I get it. I really do. Most of our tooling is so incredibly hostile to users, and our systems so opaque and mysterious, that once we find a way to explain a thing we fucking cling to it.

(I am *not* an early adopter. Those people are nuts)

But just look at the way you debug systems today. You jump from a dashboard that shows the big picture, over to another tool that lets you search for strings you pull from memory (if you're lucky and the developer added them), and jump again to try and trace a similar request...

This is all assuming a ton of best practices were followed consistently and multiple tools were paid for.

This comical set of disconnected tools and expensive, puzzling services to monitor system resources nobody understands is your BEST CASE SCENARIO

We have to get out of this state where the debugger of last resort is always the person who has been there the longest.

We act like that's inevitable, a fact of nature, but it's not. It's a side effect of a system where the only source of truth is in people's heads.

We have to get out of this state where we wouldn't accept SPOFs in our systems, heavens no, but we accept them every day in our teams -- because we have no practice interrogating production systems.

We have to get out of this state where tracing is considered yet another disconnected product to shell out for... instead of simply an extremely useful way of visualizing events in a distributed system.

(Tracing is a feature, a visualization layer, not a product -- god knows why no one but honeycomb is treating it that way. HOLD YOUR VENDORS TO A HIGHER STANDARD, PEOPLE.)

(And you wonder why you're spending so much on this shit.. when you're storing everything 5+ times 😒)

I could go on, but I'm ready to snuggle. Tldr your tools obfuscate and fight you at every turn; they were made for other purposes.

Look at how hard it is to learn. Look at how resistant software eng are to going on call. Look at how much intuition it takes to debug something.

Don't wait for everyone to master the old ways before you will let them have the easier, more intuitive tools that will actually help them do their jobs better today.

The end. 🥃🥃🥃