Thread by @mipsytipsy

Couple of "mildly antagonistic" responses to my piece on how much your o11y spend should be.

Which is good! Happy to see people engaging critically with this complex and fast-moving target. ☺️ Let's consider a few of the objections.

https://twitter.com/changelog/status/1075486679060934657

"That's too much to pay for a few dashboards!"

Agree, fully. But I'm talking about rich, full stack telemetry that lets you ask and answer any question about your large, chaotic distributed system and its emergent behaviors.

https://twitter.com/parkercloud/status/1075079467184214016?s=20

Let's be real: most of you have never experienced this kind of tooling. It hasn't been available outside FAANG until recently (hi).

Many if not most of you may not even have run systems that *demanded* this kind of observability. Scale changes everything.

Systems are getting more complex at a rate which has accelerated in the last few years, and systems observability has gotten better too -- and more necessary.

(I wouldn't pay 20-30% of my infra spend for time series graphs and dashboards either, so I actually agree with you.)

even more importantly, as @leonadato pointed out, you aren't actually paying for the o11y tools themselves; you're paying for the value of the service or business you need to understand so you can meet your SLOs. Critical difference.

https://twitter.com/leonadato/status/1075111320196915204

If your service doesn't have that much value? It doesn't make sense to spend a lot on tools that are built to help you squeeze out that third, fourth, fifth nine.

If your service isn't that complex? Then you save oodles on tools designed to help explain complex systems.

(The #1 trick for keeping your observability spend low is and always will be "simplify your god damn architecture". Microservices were the gods' greatest gift to o11y vendors like me.)

It is way way way harder to explain and understand a complex system than it is to build one. Building complex systems is relatively easy! Running them and understanding them is profoundly hard.

Other tricks to keep your bill down include

* Build something people don't care about much
* Build something where availability isn't tied to revenue
* Build something where good enough is good enough

I'm not joking. FB rode a scrap horse of php+mysql all the way to the bank.

Good observability tooling is both

a) expensive, but
b) cheaper than your other options

You can now buy these world class tools and rent these best of class engineers, which is awesome, because you could never hire them. But when you see the bill you might be like 💰💰💰🤑🤑🤑

But this is because it's surfacing the real costs which you have been silently choking on without realizing. You are paying a team of people to try and solve these problems for you, which I guarantee is more expensive than any service out there, and/or you are ..

losing 30-40% of your *entire development team's time* to technical debt. The single greatest contributor to which is *not understanding how your own systems work*, i.e. your observability sucks. (And sometimes both.)

Remember this study? stripe.com/files/reports/…

Almost nobody takes headcount or lost productivity time into account when evaluating a vendor's price, and they should. Sigh. I so wish this was de rigeur for engineering managers.

Another great piece from @leonadato on the cost of not monitoring, and why what really matters is the data or system being monitored, and how you value it. adatosystems.com/2018/12/17/the…