,
21 tweets,
5 min read
Read on Twitter
Another day, another article written about the @honeycombio-shaped hole in the world of operational tooling, without -- somehow -- ever mentioning honeycomb. 🤨 sumologic.com/blog/business-…
I am done grinding my teeth (didn't take long, loads of practice) and will instead recap it for funsies.
First off, we marvel at the growth of kubecon and nod to its intense (lol) complexity. Traditional monitoring won't work, this sounds like a job for OBSERVABILITYMAN 🦸♂️!!
First off, we marvel at the growth of kubecon and nod to its intense (lol) complexity. Traditional monitoring won't work, this sounds like a job for OBSERVABILITYMAN 🦸♂️!!
Then the article (that doesn't mention us) links to a terrific talk (that doesn't mention us), on why the old school "three pillars" model for o11y is fatally flawed.
It's a great talk, you should read the slides. schd.ws/hosted_files/k…
It's a great talk, you should read the slides. schd.ws/hosted_files/k…
Then we mention ballooning costs (yup, esp since you are paying half a dozen vendors to understand the same events slightly differently) and that tracing is a life saver in distributed systems like k8s (yup).
Now brace yourself for the greatest non sequitur leap of the new year:
Now brace yourself for the greatest non sequitur leap of the new year:
... Prometheus is king! Winning everywhere!
"But wait," I hear you asking. "Is Prometheus going to help with any of those problems, of complexity and end to end tracing and the request path? Does Prometheus even do observability? Isn't it metrics and preaggregated dashboards?"
"But wait," I hear you asking. "Is Prometheus going to help with any of those problems, of complexity and end to end tracing and the request path? Does Prometheus even do observability? Isn't it metrics and preaggregated dashboards?"
You, dear reader, have been paying attention. If you buy the highly technical, control theory-derived definition of observability that I do, then tools based on metrics (the technical definition: a number with appended tags) will never be observability tools.
Why? Because they have torn up and discarded all the connective tissue of the event before they ever write to disk.
That connective tissue is exactly what you needed to reason about the internal workings of your system as a developer. It tracks your user experience too.
That connective tissue is exactly what you needed to reason about the internal workings of your system as a developer. It tracks your user experience too.
I am not saying metrics/aggregate tools have no value! They can have loads of value. Primarily for ops use cases, like provisioning and system health.
Developers don't give a shit about system health. They care about the health of *each individual request*. Events.
Developers don't give a shit about system health. They care about the health of *each individual request*. Events.
This is the most prominent difference between monitoring (ops uses, aggregates) and observability (dev use cases, unique events).
I mean, ops doesn't give a shit about each and every request either, as long as the system is healthy and errors below SLOs.
I mean, ops doesn't give a shit about each and every request either, as long as the system is healthy and errors below SLOs.
Prometheus is a monitoring tool. It is a *great* one. It represents the peak of time series dashboard software in the wild.
My problem with it is that they claim it's more than that. Which leads to a very bad experience for users with more honeycomb-ish shaped problems.
My problem with it is that they claim it's more than that. Which leads to a very bad experience for users with more honeycomb-ish shaped problems.
Case in point.
Case in point.
Case in point.
Case in point.
Those are just from today. EVERYBODY has bit the dust on this.
And yes, I am aware that some proprietary implementations of metrics based systems do not have the same cardinality limitations, but doesn't invalidate my point -- they still aren't oriented around the event.
And yes, I am aware that some proprietary implementations of metrics based systems do not have the same cardinality limitations, but doesn't invalidate my point -- they still aren't oriented around the event.
Because once you are gathering these events, it is *trivial* to then add a field with ordering, and 💥boom💥 now you get tracing for free.
It's even better than it sounds. You don't have to double your spend. Don't have to hop from tool to tool. It just works.
It's even better than it sounds. You don't have to double your spend. Don't have to hop from tool to tool. It just works.
Tracing is just a fancy visual overlay on the rich events if you father them correctly.
Which means you can flip back and forth between exploring ("find me an example of this bug"), tracing ("now trace it"), and exploring ("who else was affected?").
Holy grail ✋
Which means you can flip back and forth between exploring ("find me an example of this bug"), tracing ("now trace it"), and exploring ("who else was affected?").
Holy grail ✋
Guess I need to do a better job of sucking up to the CNCF club for them to start acknowledging our existence. Sigh.
I'd expect better from our fellow vendors, except.. never mind, I guess this all makes sense. 🖕 just keep muddying the waters, bros
I'd expect better from our fellow vendors, except.. never mind, I guess this all makes sense. 🖕 just keep muddying the waters, bros
(But k8s is our third largest integration, and it works exactly as advertised. Just sayin.)
Final case in point:
P.S. @honeycombio has a free tier now. ICYMI
