Thread by @mipsytipsy

omg. just got this notice terminating our crappy old @signalfx infra monitoring. for using it to "build a competitive product or service".

...

there is *so* much that needs to be said here but first I need to stop crying with laughter

where to even start. hmmm

let's rewind the clock to 2016. I was a spry young thing terraforming in rage, spinning up and destroying fleets of t2.micros many times a day. I thought,

"I should have some Monitoring. Well, I hear DataDog is ok."

DataDog was indeed "ok". Until I got my bill and saw that t2.micros cost the same as an x1e.32xlarge.

AND that they charged me for something absurd based not on the number of concurrently running hosts, but the TOTAL HOSTS/MONTH.

Reader, I shit a brick. And broke up with DD.

Decided to try SFX next, bc I couldn't tell the difference between their product and DataDog's. (Still can't.)

They weren't my first pick because there was some weird territorial behavior around them. Some exFB engineers openly sneered at me and Cyen and told us to quit, etc.

When we were raising seed money we would talk about the FB/Parse/Scuba heritage of what we wanted to build, and we kept hearing "isn't that what SFX is doing? Building the FB monitoring system for everyone else?" and we'd try to explain "no, their heritage is ODS, v different"

So maybe they thought we were muddying the water for their brand or something. Hell if I know. It wasn't easy to explain bc nobody was talking about events or observability (except as a generic term for telemetry) at the time.

But fine, so I deployed SFX to my new prod.

I will say this for them, they didn't pull quite the same shenanigans with pricing that DD did.

IIRC, the $/server/month was actually just an estimate of how many writes they estimated a server should make. So I dropped the polling interval by 5x and cut my bill by 80%.

They were pretty mad about that and warned me gravely that I wouldn't get my 1sec resolution. I said well that seems pretty moot when my dashboards are consistently behind by 3 to 30 min.

(Still were, last I looked.)

(Honeycomb is almost never more than 1-3 sec behind, JFYI)

What I hadn't really thought through at the time, was that the 1sec interval is ALSO useless due to the aggregation at write time.

You can never get back to the original event from a metric, so you're reduced to debugging via interpretive dance and gut instinct.

So I think we've been giving them $800/mo for the past two years or so. Every time they try to upsell us we just tweaked our usage and considered turning it off, but that always sounded like work.

... I'm not being totally fair. SFX was useful to us in a few ways:

1) verifying ourselves. In the early days, we had no certainty that our product worked, so we would compare our results to SFX when possible.

2) paging and alerting. Honeycomb has triggers now, but it's not a *monitoring* platform, and it was nice to use some richer features.

3) and they had some great plug-ins for e.g. monitoring Kafka health, so we didn't have to develop our own. For a while we really did think we might do that.

But look. Knowing what I know now, it is *absurd* to think we are competing w/datadog, SFX, wavefront, graphite, etc.

Honeycomb is not a monitoring tool, it's a debugging tool.

It's a way for software owners to explore insanely complex/interconnected systems and explain anomalies, via instrumentation. It's about zooming way down to the nitty-gritty event level, and way out to a birds-eye view.

Monitoring is about the health of the system. 1sec intervals and time series aggregates are the appropriate tools for this!

Observability is about the health of the event, and every slice of events. Don't give a shit about the health of the service if the event can complete.

Monitoring is still very much needed for infrastructure teams. You care about trends, provisioning, disks filling up, redundancy, replication lag, etc.

Monitor those things.

But *observability* is what's needed by anyone who is writing new code, shipping new versions, or debugging their software in prod.

You need events and variable names and request context and tracing. Not monitoring.

(Yes, there is some overlap -- tho less than you'd think. It's more that data tools are all fungible and reusable for other data problems to some extent. Someone out there is using *Excel* to understand their systems, guaranteed.)

In conclusion, the only way we could conceivably be a threat to the metrics shops is if they are trying to mislead their customers into thinking they are a one stop shop, all you need to understand the software you build as well as the infra you run.

But that seems far-fetched.

And I am sad to bear witness to the demise of our @signalfx account, if only bc now i need a new one.

Is there a hosted metrics provider who feels less threatened by us than SFX apparently does, and would you be interested in swapping accounts on our respective services? 📈🌈💕

(Ok full transparency I do think over the next 5-10 years the categories monitoring/metrics, logging, and APM will collapse into one o11y category. The distinctions only exist bc of decisions made based on hardware resource scarcity, which no longer true due to plunging costs.)

(But in the long run we all compete, like why is a book seller competing with a search engine? So let's worry about that when we have the problems of massive success ok?)

By the way, while I'm talking smack on others in the space -- which I really do try to avoid -- i'd also like to call out @stevescalyr and @scalyr for the opposite.

They have been unfailingly kind, supportive and helpful, even back when it seemed plausible we might compete.

I think @SteveScalyr tried to recruit me when I was looking in to the space. We stayed in touch; I later confessed abashedly that I wanted to start my own thing instead.

He offered us his help and advice and time, without even missing a beat.

Believe me, we took him up on it. He said his philosophy has always been ideas are cheap, most startups fail, we should all just help each other out.

I'm nowhere as nice as him but I agree. So if you need a logging solution, or want a job, you cant do better than @scalyr.