Charity Majors Profile picture
cofounder/CTO @honeycombio, co-author of Observability Engineering and Database Reliability Engineering. I test in production and so do you. 🐝🏳️‍🌈🦄
Marc Byfield Profile picture Michael Hood Profile picture Nat Welch Profile picture Ragan Profile picture John Howard Profile picture 43 subscribed
Apr 22 12 tweets 3 min read
The question is, how can you interview and screen for engineers who care about the business and want to help build it, engineers who respect sales, marketing and other functions as their peers and equals?

It's a great question!! I have ideas, but would love to hear from others. I said "question", but there are actually two: 1) how to hire engineers who are motivated by solving business problems and 2) aren't engineering supremacists.

They are not *unrelated*, but they are different things.charity.wtf/2022/01/20/how…
Mar 22 7 tweets 2 min read
Ooooohhh boy, this is a terrific question. I have written two closely related pieces,

* for engineers interviewing at a new company, on how to sniff out bad management culture:

* how to tell if the co is rotten on the inside: charity.wtf/2021/02/19/que…
charity.wtf/2022/01/29/how…
But both of those were written from the perspective of the engineer/interviewee, not the interviewer. The dynamic is different, for sure. 🤔

I would probably start by asking them why they became a manager, why they enjoy the job (if they do). (Softballs)
Feb 14 12 tweets 3 min read
Pro tip: any time you see someone confidently opining on what all good CTOs know or do, it is ✨bullshit✨

There is no stock template for CTO, or default set of expectations or responsibilities. It stands alone among the C-levels in that good ones are all over the freaking map. This may not hold true for publicly traded companies. But in my experience, a good CTO can be:

* over all of R&D
* over engineering, like a VP eng
* like a principal eng or architect
* team lead for special projects
* a great senior programmer

(continued... 👉)
Jan 22 7 tweets 2 min read
Yeah, this is a fair caveat. If you're already a decent senior engineer and manager, it's kind of possible to split your attention between managing a small team and writing code.

But you aren't going to improve at either skill set. Those cycles get devoured by context switching. Tech lead managers ("TLMs") are a mistake we make over and over in this industry. I've written about this a bit, but the definitive post was written by @Lethain.



Instead of being the best of both worlds, TLMs are poorly equipped to do either.lethain.com/tech-lead-mana…
Jan 8 12 tweets 3 min read
My coworker @suchwinston wrote a terrific piece on burnout before the break:

There's a reason why burnout and work/life balance are such evergreen topics, and it's not actually because the world is so terribly harsh and everyone is criminally overworked.honeycomb.io/blog/product-m… Just to be clear: some places *are* awful, and some people *are* criminally overworked. But burnout and work/life balance are an issue for everyone, not just those people.

I think this is bc there is no real "solution". Each of us has to find and maintain our own equilibrium.
Dec 29, 2023 6 tweets 2 min read
This is an astute point. For all the ink that has been spilled about what observability is or is not, or how generation 2.0 differs from 1.0, it's actually quite simple.

"Observability" was coined to denote the emergence of ✨high cardinality✨ support in telemetry and tooling. Cardinality, for those new to my feed 🤣 refers to the number of unique items in a set. Gender drop-down with three options? Low cardinality. Gender field you can write to? Much, much higher cardinality.

Metrics can't do high cardinality data. A metric can only be a number.
Dec 22, 2023 32 tweets 6 min read
About two months ago I wrote this thread about how we lost the battle to define ✨observability✨ -- to give it a real, specific, falsifiable technical definition, distinct from monitoring or telemetry.

I complained, I argued, I grieved...and now I'm over it. SO over it. 🙄 In fact, I've come all the way around: I AGREE.

It is less useful for observability to stand for a specific set of technical capabilities and outcomes (explorability, high cardinality etc).

It is more useful, and less confusing, to define observability as a property of systems.
Dec 2, 2023 6 tweets 2 min read
A couple months back this show called "Suits" was having a Moment, so I made it my falling-asleep show for a while.

As a show, it's ok. Pretty formulaic, but that should be *good* for winding down. It took me a while to untangle why it was getting under my skin so badly. Things that really bothered me about Suits:

* The people in charge refuse to tell anyone else what they are doing, or why they are doing it

* They rarely make requests of their underlings, and NEVER share context or background info; they bark orders and demand total obedience.
Dec 2, 2023 5 tweets 2 min read
This is a hard topic. ☺️ this statement in particular is one I might reject out of some mouths, but coming from @maggiepint, I know what she means. And yeah.

It is hard to hold a nuanced position on a complex topic, and harder still to convey. The dominant problem in tech is discomfort with/denial of feelings, so naturally the biggest backlash is about making it okay to be a human with physical and mental needs, normal emotional range, and family that needs attention.

But you can swing too far that way too.
Nov 27, 2023 7 tweets 2 min read
Yes. Being a great engineering manager who cares about people is exhausting.

Professionalism itself is a form of mild dehumanization; you and I set our identities and personal selves to the side, and interact with each other's professional functional self. We become cogs. 🙃 A manager needs enough emotional distance from their team to be able to make a decision because it is what's best for the system, not because it would make their team happiest.

You need to learn to use people like resources, but not lose touch with your humanity.. or theirs.
Oct 30, 2023 9 tweets 2 min read
So, we lost the battle to define observability. You know it, I know it. Observability was supposed to *mean* something, and in the early days, it did.

"Observability" once meant the kind of exploratory, open ended investigation our systems increasingly demand. And because you can't get it using old fashioned tools, "observability" also stood for all its prerequisites:

* high cardinality
* high dimensionality
* traceability
* structured log events
* rich context
* end to end instrumentation
* from the user's perspective -- each user's
Oct 27, 2023 8 tweets 2 min read
✨Happy Friday!✨ I'd like to take a sec to debunk a popular myth about the "deploy on Fridays" movement. Many seem to think we mean everyone should be deploying code right up til quitting time, every Friday.

This is not the point at all. This is NOT a great idea for most shops. I can see why people might be confused, because I would say that people shouldn't be *scared* of shipping on Friday. If there's some business or technical reason that occasionally keeps you writing code and shipping it all day Friday, nobody should run screaming for the hills.
Sep 26, 2023 8 tweets 2 min read
Over the last six months, SumoLogic, NewRelic, and Splunk have all been sold off to private equity firms (or noted tech graveyard Cisco).

I think it's harder than people realize to innovate for the next generation of observability when you were built to solve the last one. Observability is a UI/UX problem, an instrumentation problem, a scale problem... but it is also very much a data problem.

The shape of the data and the way that you store it will confine your product decisions for years to come, in ways that can be non obvious at the outset.
Aug 31, 2023 8 tweets 3 min read
This morning I recorded a new podcast with @theburningmonk, which was super fun. (coming soon!)

Yan pushed me on the question of Friday deploys. He says some people think it means that you should be able to ship every Friday at 4:59pm, then cruise on out the door. No?

Noooo. 🤦‍♀️ First of all: nobody should ever "deploy and walk out the door". Not on any day, not any time of any day. If you deploy your code at 10 am Tuesday and walk out the door, you're an asshole.

Your job writing code is not done until you've watched it run in production. Period.
Aug 27, 2023 45 tweets 21 min read
I just got back from @fintechdevcon in Austin. It was a super fun trip (notwithstanding the 105 degree air 🥵).

I gave a new talk while I was there, on why compliance and regulatory standards are ✨not✨ incompatible with modern development best practices. Image "Episode 01010010: Regulatory Compliance Is Not The Enemy

The fintech galaxy is ridden with cargo cults and myths about how regulatory standards prevent them from building software using modern best practices..."

(I was going to open with this vid, but ran out of time for A/V)
Aug 19, 2023 9 tweets 2 min read
Oh man, I also meant to ask... do people still *do* this? Run massive logging, indexing and searching clusters, I mean.

How do you justify the labor and opportunity costs, in an era where logs are a commodity and hosted services are dirt cheap? Serious question... I'd like to understand.
Jul 6, 2023 13 tweets 3 min read
Good people management has been very top of mind for me lately. 🤔 Honeycomb is big enough now that we are starting to develop our own "manager training" materials (except not just for managers, and more like an...immersive experiment?) There are so many big, hard, shitty lessons you have to learn that I don't think I've ever heard anyone talk about.

Or if they do, it's in the bloodless, studiously bland voice of the business aisle. "To be a good leader, you must build trust"?? NO SHIT Sherlock.
Jun 19, 2023 9 tweets 2 min read
Many new engineering managers come to the job full of frustration at the shitty management they have endured and/or witnessed, and determined not to inflict the same shit on their teams.

And they don't! 💕

Instead, they make new mistakes...like these.

charity.wtf/2023/06/19/thr… Rookie mistake number one is thinking that all there is to management is coaching and developing your team.

That's a big part of the job, of course, but you also have responsibilities to the org, to your peers, to upper management...and to the success of the business.
May 19, 2023 10 tweets 2 min read
I have ranted about the absurdity of the term "full stack engineer" (I mean, show me the web dev who also writes their own device drivers 🙄) but I'd like to recant.

Annoying it may be, but it's the term we've got to describe the most significant shift in eng roles since devops. I am an extremely literal person, and literally speaking, nobody can be a "full stack" engineer. The idea is a ridiculous one. There's too much stack!

But that's not what people mean when they say it. They mean, "I'm not just a frontend or backend engineer. I span boundaries."
May 3, 2023 23 tweets 5 min read
This might already count as old news, since apparently over 20% of our users have already tried this, but... This morning we launched the first ever natural language interface for observability data, using generative AI and chatgpt. 🙃🐝🎉✨ Our generative AI based Query Assistant is available to all honeycomb users, including free tier users, for free.

@_cartermp wrote a great blog post on what we've built and why, and tips on crafting great prompts (prompt engineering is the new hawt skill) honeycomb.io/blog/introduci…
May 1, 2023 4 tweets 2 min read
I just wrote a new blog post, called “Choose Boring T̶e̶c̶h̶n̶o̶l̶o̶g̶y̶ Culture”, on the difference between formal/org culture and informal/bottoms-up culture, what *actually* makes work fun, and where to invest your innovation tokens.

charity.wtf/2023/05/01/cho… The problem with writing blog posts is they have a way of turning into lists of other blog posts to write. 🙈

I was making the point that nobody actually wants an “exciting” company culture, because excitement usually proceeds from dysfunction… Image