Thread by @mipsytipsy

lol I completely forgot about the time that @signalfx cut us off as customers in good standing, with no warning and no explanation.

Remember that? it was like 5 am on a Tuesday, and all our alerts just started firing at once.

https://twitter.com/mipsytipsy/status/1164538938784014337

We start scrambling -- this is *extremely* unusual for us, that we get several alarms at once -- and soon realize that @signalfx just switched us off.

Our monitoring company

Turned our service

Off

In the wee hours.

Out of fucking spite, for all we can tell. ¯\_(ツ)_/¯

Now listen, i may rag on the humble metric once in a while, but metrics are the right tool for many jobs and I recommend people use them for several workloads. As did we.

Those workloads include:

a) low level infra stuff
b) long lived historical trends
c) capacity planning
d) health of the system in aggregate
e) counters
f) anything involving java /jmx, for now

So, we were definitely relying on them for our kafka stats and stuff. It was deeply uncool to pull the cord.

Fortunately, the lovely folks at @WavefrontHQ were much less insecure. They promptly hooked us up, and we have been shipping our metrics to them ever since.

(Thanks wavefront! 💕)

You might be wondering if wavefront reminded me about this sordid little episode. No, they did not.

SignalFX employees reminded me. 😕

Apparently there was some internal angst at the time about why the leadership chose to shut us out rather than partner with us.

I knew nothing of this at the time -- disclaimer -- and I still have nothing but hearsay. Who knows. But.

I remember going down to introduce ourselves to the sfx leaders in the early days. We just wanted to say hi and make friends.

🥺 We even drove to the South Bay. 🥺

For those who don't know, sfx and hny share DNA. Facebook had two main systems for devs to instrument their code: ODS and scuba.

ODS was the metrics behemoth, a type we all know. Scuba was the weird junior sibling; hacked together a decade ago in the darkest days of mysql.

Ask any Facebook dev, and they will tell you they needed both.

* ODS metrics for cheap counters and historical statistics and aggregates.
* Scuba for *actually understanding their code and systems*.

You need both. (But I've only ever seen devs get misty over missing scuba.)

Signalfx, of course, is the descendent of ODS, and honeycomb was inspired by scuba. We naively thought there might be room for collaboration.

It was the most awkward, hostile meeting of my life. 🙃 They pulled a bunch of weird power moves to make us feel small.

Told us it was a solved problem, that there was no reason for us to exist. It was confusing and awkward.

At one point I asked how they were different from datadog. If looks could kill...🥶🤯

In retrospect I realized they were probably angry with us bc at the time their whole line was "signalfx == Facebook metrics", so we were muddying the waters for their message just by existing and mentioning other tools.

Anyway. The sfx founders may have been giant assholes to us, and I do think they missed out by not collaborating with us.

But I am so happy for the engineers and other folks who got some of that phat logs gold. Congrats. 👏

Oh, and times I have missed the sfx dashboards: 0.

(Sorry kids, they are pretty much a commodity now: an awful, awful legacy commodity that shrinks your brain and makes you stupid. Friends don't let friends drive non interactive dashboards.)