Thread by @mipsytipsy

"there are spikes in our graphs all the time, we used to just ignore them. now, with honeycomb, we just break down or use bubbleup and go 'oh, it's just slow for that one user' or 'just that endpoint is spiking'. it's so easy.

we used to guess all the time, but now we /know/"

if there's one quote we hear from users over and over and over again, it's that one.

"we used to guess. now we know."
"i used to run crazy sed/awk/histogram scripts, now i can just look"
"we used to sweep it under the rug or it would eat up all day; now we don't have to."

this also reminds me that i have done a poor job of alerting y'all to one of the most exciting things we've ever shipped, bubbleup (formerly known as drilldown).

everyone always used to ask, can you tell me what to look at? and i would sigh wearily and say no.

until now.

we think of it as the 'core analysis loop' of debugging:

1) formulate a hypothesis about what variable could be causing a behavior
2) break down by that variable to test its relevancy
3) check, repeat as necessary

honeycomb.io/blog/heatmaps-… @FisherDanyel awesome post on this

@FisherDanyel the reason that honeycomb encourages you to ship off rich events, packed with dense context, is to give you as many different ways of asking this question as possible.

take a spike of errors. you want to know if they were mostly to a shard, a node, a pod, a user, an ip addr...

@FisherDanyel if you're using a metrics-based tool, you have an error counter, but you have no way of working backwards to "what 1..n dimensions do all these errors have in common?"

with structured events you can do this, either by intuition or brute force.

@FisherDanyel you know, by now, my feelings on intuition-based systems for debugging distributed systems. they are to be deprecated, let's put it that way. but brute force isn't appealing either.

enter bubbleup. honeycomb.io/blog/diving-in…

@FisherDanyel any time you select an area on the heatmap, we zoom out and pre-compute histograms for every dimension in that area, and compare it to baseline values.

your eye is *immediately* drawn to whatever outliers exist, however many of them there are. it's like fucking magic.

@FisherDanyel my laptop monitor is too small for me to screenshot the full honeycomb ui, but check out @cyen's little animated gif of using the feature:

https://twitter.com/cyen/status/1065355541554155520?lang=en

see how easy it is to spot the problems? pure sorcery.

@FisherDanyel @cyen as our sales engineer said, with what i *think* was mostly joy: "it hardly takes any time to demo honeycomb now, there's nothing to debug. you just look, it's there. so obvious."

this is why i keep pounding on about using the beelines and structuring your goddamn data.

@FisherDanyel @cyen if you're a honeycomb user you can use the beelines. it makes everything magical for you, it "just works". all the batching and blobbing of information on a per request/per service basis.

if you're not a honeycomb user, *you should still do this.*

*this still matters.*

@FisherDanyel @cyen all our beelines are open source; if you want to check out how we're doing it and mod it up for yourselves, be my guest.

if you are adopting microservices, know that you really cannot understand how your system works unless you pack the data up this way.

github.com/honeycombio/

@FisherDanyel @cyen any vendor who tells you that you can give them millions of dollars, make zero changes to your code, and achieve observability nirvana

is

lying

to

you.

and if you bought it, you were the sucker who knew better deep down and did it anyway, weren't you?

(i'm sorry)

@FisherDanyel @cyen (blergh, still no way to untag people you accidentally dragged into a twitter thread, eh? my apologies to fisherdanyel and cyen. 🙏)