Thread by @mipsytipsy

oh dear, it seems i've inadvertently helped promulgate a myth. kittens,

* honeycomb does not sample
* honeycomb does not sample
* honeycomb does not sample
* honeycomb has NEVER sampled

we have written some helper libs that let you do some cool client-side dynamic sampling...

... which you could just as easily use with or without the rest of our (best-in-class) application auto-instrumentation, BOTH of which you could AND PROBABLY SHOULD use even if you are not using any honeycomb whatsoever.

because sampling is FUCKING AWESOME.

because all data is not created equally, and not all debug spew is worth paying to store

because we can do a damn good job of predicting the data that is definitely going to be junk/not cost-effective (health checks anyone?)

the one thing we DO do in honeycomb, is perceive the sample rate of any given event, so we can do the math and draw the lines correctly.

so that if you're sampling 200 OK requests to the root domain at 20-to-1, it doesn't look like you have 1/20th as much traffic as you do.

if you aren't sure why, sampling "just makes you feel weird" -- it's not your fault. you've been soaking up marketing messages from logs vendors for decades, as they try to scare you into thinking every trash line is sacred.

if you're interested in a very reasonable, approachable corrective to this fearmongering, try one of liz's recent talks on why sampling is both inevitable and wonderful.

usenix.org/conference/sre…

but as for honeycomb, no, we don't do any sampling. you are just as welcome to ship all your trash into honeycomb (and pay us for it) as you are into any other vendor.

our incentives are, in theory, the same as theirs -- if you choose not to sample, you pay us more. WAY MORE!

you could argue that we were short sighted, and should have *not* built knobs for cost-effectiveness or tried to educate folks about sampling.

you might even be right. i didn't realize at the time what an industry outlier we would be. but it would have felt very deceptive. 😕