Profile picture
Charity Majors @mipsytipsy
, 14 tweets, 4 min read Read on Twitter
Couple of "mildly antagonistic" responses to my piece on how much your o11y spend should be.

Which is good! Happy to see people engaging critically with this complex and fast-moving target. ☺️ Let's consider a few of the objections.
"That's too much to pay for a few dashboards!"

Agree, fully. But I'm talking about rich, full stack telemetry that lets you ask and answer any question about your large, chaotic distributed system and its emergent behaviors.
Let's be real: most of you have never experienced this kind of tooling. It hasn't been available outside FAANG until recently (hi).

Many if not most of you may not even have run systems that *demanded* this kind of observability. Scale changes everything.
Systems are getting more complex at a rate which has accelerated in the last few years, and systems observability has gotten better too -- and more necessary.

(I wouldn't pay 20-30% of my infra spend for time series graphs and dashboards either, so I actually agree with you.)
even more importantly, as @leonadato pointed out, you aren't actually paying for the o11y tools themselves; you're paying for the value of the service or business you need to understand so you can meet your SLOs. Critical difference.
If your service doesn't have that much value? It doesn't make sense to spend a lot on tools that are built to help you squeeze out that third, fourth, fifth nine.

If your service isn't that complex? Then you save oodles on tools designed to help explain complex systems.
(The #1 trick for keeping your observability spend low is and always will be "simplify your god damn architecture". Microservices were the gods' greatest gift to o11y vendors like me.)
It is way way way harder to explain and understand a complex system than it is to build one. Building complex systems is relatively easy! Running them and understanding them is profoundly hard.
Other tricks to keep your bill down include

* Build something people don't care about much
* Build something where availability isn't tied to revenue
* Build something where good enough is good enough

I'm not joking. FB rode a scrap horse of php+mysql all the way to the bank.
Good observability tooling is both

a) expensive, but
b) cheaper than your other options

You can now buy these world class tools and rent these best of class engineers, which is awesome, because you could never hire them. But when you see the bill you might be like 💰💰💰🤑🤑🤑
But this is because it's surfacing the real costs which you have been silently choking on without realizing. You are paying a team of people to try and solve these problems for you, which I guarantee is more expensive than any service out there, and/or you are ..
losing 30-40% of your *entire development team's time* to technical debt. The single greatest contributor to which is *not understanding how your own systems work*, i.e. your observability sucks. (And sometimes both.)

Remember this study? stripe.com/files/reports/…
Almost nobody takes headcount or lost productivity time into account when evaluating a vendor's price, and they should. Sigh. I so wish this was de rigeur for engineering managers.
Another great piece from @leonadato on the cost of not monitoring, and why what really matters is the data or system being monitored, and how you value it. adatosystems.com/2018/12/17/the…
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Charity Majors
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member and get exclusive features!

Premium member ($30.00/year)

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!