Thread by @_cartermp

This is my romantic and idealistic take one why I think #OpenTelemetry matters 🧵

(ah fuck, I'm doing one of these now)

Modern software services are really complex and there's no easy button you can press to dig into problems in these systems. I wish it were easier, but magic isn't real and santa claus doesn't exist, so therefore you have to put some work into ✨Instrumenting your systems✨

Creating distributed traces, metrics, logs, whatever ... this is a tax. It is the tax you pay to get a sufficient degree of Observability. And Observability is the fancy word for "debugging your systems and understanding what they do in the real world".

As such, you should only ever pay this tax once*. But how? OpenTelemetry!

opentelemetry.io

* data export code will change based on where you send it, but it is very little code

OpenTelemetry is supported by many vendors, my employer included. We spend time and money making it better because we believe that our products should compete on how effectively they save YOU time and money. And if we aren't cutting it, you should be able to easily switch!

But it's also not just about being able to shop around between Honeycomb, Lightstep, Splunk, New Relic, Datadog, etc. more easily.

Your use of OpenTelemetry forces vendors to make their products ✨better✨.

Example time!

Honeycomb doesn't handle Span Events ideally. They are treated as individual events, billed separately from the "event" that represents the span they're on. A span with 10 Span Events will count as 11 billable Honeycomb events today.

opentelemetry.io/docs/instrumen…

This isn't bad, though! By being a Honeycomb Event from the data store's perspective, Span Events are queryable entities. And yes, I looked at the data, customers indeed query against Span Events, so it's a model that can be useful.

But! Automatic instrumentation modules exist, and some of 'em are too eager to go wild with creating tons of Span Events that aren't actually useful to query. We had a customer inadvertently make their billable events go 📈 this way. Yikes! 😬

opentelemetry.io/docs/instrumen…

So what'd we do?

1. Updated our own systems to count up spans, span events, and span links independently
2. Identify the (small) number of customers with a very high volume of non-span events
3. Work with them to make their bill reasonable

(cont...)

4. Come up with no less than 7 proposed solutions to make this not be a problem in the future
5. Put it on the roadmap to work out the right solution and implement it
6. In the interim, monitor for any new customers encroaching a bad bill so we can work with 'em too

Let's say that instead, we were shitty, did nothing and told customers to just "deal with it".

Well, it would take them an afternoon to start sending data to a competitor who handles Span Events more economically.

Now THAT'S some damn accountability.

Open standards mean that vendors cannot afford to have wonky behavior in their products and be shitty about it. They force vendors, even if the vendors are all nice and cuddly like we at Honeycomb are, to buckle up and fix stuff for customers or risk losing business.

As much as I love companies that act ethically, at the end of the day, I want to live in a world where we can hold them accountable to their obligations.

And that's ultimately why OpenTelemetry matters. It's an accountability system built into an Observability framework.

So yes, you should look into ripping out your beeline, ddtrace, xray, AppInsights, etc. code and replacing it with OpenTelemetry in the long run. And if there's gaps, file bugs!

Check out an SDK for your language here: opentelemetry.io/docs/instrumen…