now, if you haven't been paying attention, we have racked up quite a library of useful material over the past year at @honeycombio. 🐝

our first white paper is the best answer you're gonna find anywhere to the question: ✨what is observability?✨ honeycomb.io/resources/guid…
@honeycombio our second white paper or ebook was called "observability for developers", and it focused on instrumentation: how to gather data at the right level of granularity to achieve the o11y goal of 'ask any question, without having to ship new code'.

honeycomb.io/resources/guid…
@honeycombio we then shipped a guide on tracing. we believe the events you are already gathering to debug your systems should be *plenty*, without having to store all that data (& pay for storing that data) in a completely new silo.

honeycomb.io/resources/alwa…
@honeycombio if you were at @o11ycon last year, you know we broke out into a bunch of small expert groups to share knowledge and best practices around observability. the findings of that skill share are gathered for posterity here:

honeycomb.io/resources/o11y…
@honeycombio @o11ycon oh. whoops. the tracing link above is to our tracing *webinar*. this is the guide to tracing with honeycomb-style events (or opentracing events):

honeycomb.io/resources/dist…
@honeycombio @o11ycon we then published a guide for business decision-makers to help them see why observability is a key cornerstone for cutting down on technical debt, improving MTTR, managing systems with fewer humans, devoting more time to creative labor, etc.

honeycomb.io/wp-content/upl…
@honeycombio @o11ycon next we published an intro to best practices with sampling. we as an industry have lost these muscles, and we *must* regain them if we are going to run systems at scale.

aka "if you pay your logging vendor too much, start here" 👉 honeycomb.io/wp-content/upl…
@honeycombio @o11ycon speaking of which, how much *should* you be spending on observability? my rule of thumb has always been, o11y should total 10-30% of your infra spend. but if you want to know what the professionals say, we've got a whitepaper for that too.

honeycomb.io/resources/calc…
@honeycombio @o11ycon we got a lot of questions about how (or if) someone should use us if they already have various tools. so we wrote up a paper on where we fit in the general landscape, and what we coexist nicely with or replace.

honeycomb.io/wp-content/upl…
@honeycombio @o11ycon some of these are signup gated, some are not; echo complaints >> /dev/null.

the amazing, unstoppable force behind these guides is primarily the magnificent @djpiebob, who came to honeycomb after a decade at splunk.

she walked in the door, and poof -- ✨professionalism✨
@honeycombio @o11ycon @djpiebob but honeycomb has never been just about building another dev tool or ops tool. we're on a mission ... to humanize the profession of software development.

we believe that tools change lives. we believe that data changes lives.
@honeycombio @o11ycon @djpiebob we believe that people genuinely want to do the right thing, and usually WILL do the right thing if they know what that is and have the power to do so.

our profession has grown inhumane as we have grown increasingly disconnected from our users.
@honeycombio @o11ycon @djpiebob it is unacceptable for 40% of our software engineering cycles to get drowned under in tech debt.

it is unacceptable for on call to be life-impacting, sleep cycle destroying drudgery. it is *equally* unacceptable to ask another team to suffer on your behalf.
@honeycombio @o11ycon @djpiebob observability has long been a missing link in the quest for humane, sustainable careers in tech.

without observability, you can't consistently align your engineering incentives with user experience, nor can you consistently make progress. you'll just be guessing.
@honeycombio @o11ycon @djpiebob but it's not enough on its own, either. you have to shiift the focus from the individual performer to the group, identify unique pathologies and craft custom solutions to grow your team ever closer to functional and happy.

which brings us to our latest, the "maturity model".
this paper condenses a lot of what @lizthegrey and i know about building humane, high performing teams and aligning the incentives of engineers and users, and how these goals intersect with observability.

we'd love your feedback 💛🐝🖤
honeycomb.io/wp-content/upl…
@lizthegrey ... and now i've got my replacement pinned tweet. 👆woot!

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Charity Majors

Charity Majors Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @mipsytipsy

Feb 19
Let's talk about OpenTelemetry, or "OTel", as the kids like to call it.

I remember emitting sooo many frustrated twitter rants back in 2017-2018 about how *behind* we were as an industry when it comes to standards for instrumentation and logging.

Then OTel shows up.
For those of you who have been living under a rock, OTel is an open standard for generating, collecting, and exporting telemetry in a vendor agnostic way.

Before OTel, every vendor had its own libraries, and switching (or trying out) new vendors was a *bitch*.
Yeah, it's a bit more complicated to set up than your standard printf or logging library, but it also adds more discipline and convenience around things like tracing and the sort of arbitrarily-wide structured data blobs (bundled per request, per event) that o11y requires.
Read 15 tweets
Feb 17
I want to give this a slightly longer treatment. ☺️ (Gergely and I *just* talked about it, so it's all rustling around in my head.)

I think it's a ✨great✨ idea for every engineer to spend at least a couple years at both a big company and a startup (series B or earlier).
It's hard to formulate career goals in your first decade or so as an engineer; there is just SO MUCH to learn. Most of us just kinda wing it.

But this is a goal that I think will serve you well: do a tour of duty at a startup and another at a bigco, in your first 10y as an eng.
Besides the obvious benefits of knowing how to operate in two domains, it also prevents you from reaching premature seniority. (charity.wtf/2020/11/01/que…)

The best gift you can give your future self is the habit of regularly returning to the well to learn, feeling like a beginner.
Read 20 tweets
Feb 10
Several people asked this. It's a good question! I will share my thoughts, but I am certainly not religious about this. You should do what works for you and your teams and their workflows. 📈🥂☺️
1) "assuming you have good deduplication"... can a pretty big assumption. You never want to be in a situation where you spend more time tweaking dupe, retry, re-alert thresholds than fixing the problem.
2) having to remember to go futz with a ticket after every little thing feels like a lot of busywork. You've already committed some code, mentioned it in #ops or wherever, and now you have to go paste all that information into a task (or many tasks) too?
Read 12 tweets
Feb 9
a caviar-quality rant on deployment, security, testing ... actually more like five rants stuffed into a single trenchcoat. via @beajammingh

mumble.org.uk/blog/2022/02/0…
@beajammingh the title particularly caught my eye. for the past month or two i've been sitting on a rant about how i no longer associate the term "devops"** with modern problems, but with fighting the last war.

** infinitely malleable as it may be
yes, if you have massive software engineering teams and operations teams and they are all siloed off from each other, then you should be breaking down (i can't even say it, the phrase is so annoying) ... stuff.

but this is a temporary stage, right? a bridge to a better world.
Read 18 tweets
Feb 9
I've done a lot of yowling about high cardinality -- what it is, why you can't have observability without it.

I haven't made nearly as much noise about ✨high dimensionality✨. Which is unfortunate, because it is every bit as fundamental to true observability. Let's fix this!
If you accept my definition of observability (the ability to understand any unknown system state just by asking questions from the outside; it's all about the unknown-unknowns) then you understand why o11y is built on building blocks of arbitrarily-wide structured data blobs.
If you want to brush up on any of this, here are some links on observability:

* honeycomb.io/blog/so-you-wa…
* thenewstack.io/observability-…
* charity.wtf/2020/03/03/obs…

and on wide events:

* charity.wtf/2019/02/05/log…
* kislayverma.com/programming/pu…
Read 16 tweets
Feb 6
Close! "If you're considering replacing $(working tool) with $(different tool for same function), don't do it unless you expect a 10x productivity improvement"

cvs to git? ✅
mysql to postgres? ❌
puppet to chef? ❌
redhat to ubuntu? ❌
The costs of ripping and replacing, training humans, updating references and docs, the overhead of managing two systems in the meantime, etc -- are so high that otherwise you are likely better off investing that time in making the existing solution work for you.
Of course, every situation is unique. And the interesting conversations are usually around where that 10x break-even point will be.

The big one of the past half-decade has been when to move from virtualization to containerization.
Read 12 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

:(