, 14 tweets, 7 min read Read on Twitter
"there are spikes in our graphs all the time, we used to just ignore them. now, with honeycomb, we just break down or use bubbleup and go 'oh, it's just slow for that one user' or 'just that endpoint is spiking'. it's so easy.

we used to guess all the time, but now we /know/"
if there's one quote we hear from users over and over and over again, it's that one.

"we used to guess. now we know."
"i used to run crazy sed/awk/histogram scripts, now i can just look"
"we used to sweep it under the rug or it would eat up all day; now we don't have to."
this also reminds me that i have done a poor job of alerting y'all to one of the most exciting things we've ever shipped, bubbleup (formerly known as drilldown).

everyone always used to ask, can you tell me what to look at? and i would sigh wearily and say no.

until now.
we think of it as the 'core analysis loop' of debugging:

1) formulate a hypothesis about what variable could be causing a behavior
2) break down by that variable to test its relevancy
3) check, repeat as necessary

honeycomb.io/blog/heatmaps-… @FisherDanyel awesome post on this
@FisherDanyel the reason that honeycomb encourages you to ship off rich events, packed with dense context, is to give you as many different ways of asking this question as possible.

take a spike of errors. you want to know if they were mostly to a shard, a node, a pod, a user, an ip addr...
@FisherDanyel if you're using a metrics-based tool, you have an error counter, but you have no way of working backwards to "what 1..n dimensions do all these errors have in common?"

with structured events you can do this, either by intuition or brute force.
@FisherDanyel you know, by now, my feelings on intuition-based systems for debugging distributed systems. they are to be deprecated, let's put it that way. but brute force isn't appealing either.

enter bubbleup. honeycomb.io/blog/diving-in…
@FisherDanyel any time you select an area on the heatmap, we zoom out and pre-compute histograms for every dimension in that area, and compare it to baseline values.

your eye is *immediately* drawn to whatever outliers exist, however many of them there are. it's like fucking magic.
@FisherDanyel my laptop monitor is too small for me to screenshot the full honeycomb ui, but check out @cyen's little animated gif of using the feature:

see how easy it is to spot the problems? pure sorcery.
@FisherDanyel @cyen as our sales engineer said, with what i *think* was mostly joy: "it hardly takes any time to demo honeycomb now, there's nothing to debug. you just look, it's there. so obvious."

this is why i keep pounding on about using the beelines and structuring your goddamn data.
@FisherDanyel @cyen if you're a honeycomb user you can use the beelines. it makes everything magical for you, it "just works". all the batching and blobbing of information on a per request/per service basis.

if you're not a honeycomb user, *you should still do this.*

*this still matters.*
@FisherDanyel @cyen all our beelines are open source; if you want to check out how we're doing it and mod it up for yourselves, be my guest.

if you are adopting microservices, know that you really cannot understand how your system works unless you pack the data up this way.

github.com/honeycombio/
@FisherDanyel @cyen any vendor who tells you that you can give them millions of dollars, make zero changes to your code, and achieve observability nirvana

is

lying

to

you.

and if you bought it, you were the sucker who knew better deep down and did it anyway, weren't you?

(i'm sorry)
@FisherDanyel @cyen (blergh, still no way to untag people you accidentally dragged into a twitter thread, eh? my apologies to fisherdanyel and cyen. 🙏)
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Charity Majors
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!