Charity Majors Profile picture
cofounder/CTO @honeycombio, co-author of Observability Engineering and Database Reliability Engineering. I test in production and so do you. 🐝🏳️‍🌈🦄
Marc Byfield Profile picture Michael Hood Profile picture Nat Welch Profile picture Ragan Profile picture John Howard Profile picture 44 subscribed
Jun 25 5 tweets 2 min read
Well, I for one am not past this bullshit by now. ☺️ EMs who do some hands on engineering are better EMs.

Forbidding EMs from touching code at all is almost as silly and counterproductive as telling EMs that writing and shipping code is a core function of their role. I say "almost" because if I had to choose one or the other, I would choose the clarity of "EMs responsible for team outcomes, SWEs responsible for technical outcomes" over the muddle of holding EMs responsible for everything and splitting their focus between people and code.
Jun 10 15 tweets 4 min read
I have a new piece up. It's a bit of a rant, even for me, so buckle in.

A lot of "thought leaders" have been making their mortgages lately off of bits on how AI is going to replace software engineers, particularly entry-level engineers.

stackoverflow.blog/2024/06/10/gen…
This is a dumb idea. It bespeaks a wealth of misunderstanding about what it means to be an engineer and write code, and what is valuable and hard about software systems.

But even really dumb, damaging ideas can weasel into people's heads if you repeat them blindly enough times.
Apr 28 26 tweets 5 min read
In 2023 we saw several rounds of quality conversation around engineering productivity, thanks to McKinsey, @GergelyOrosz and @KentBeck and others.

It moved the industry forwards. 🙌 But it also felt fairly inside baseball to me. Deeply technical, lots of metrics. It felt, to me, like those participating were stepping very cautiously around a few of the third rails Jaana just tripped over. (💜)

"Work-life balance"
"Working hard vs working smart"
"Meritocracy"

The intersection of company tech cultures and expectations and performance.
Apr 22 12 tweets 3 min read
The question is, how can you interview and screen for engineers who care about the business and want to help build it, engineers who respect sales, marketing and other functions as their peers and equals?

It's a great question!! I have ideas, but would love to hear from others. I said "question", but there are actually two: 1) how to hire engineers who are motivated by solving business problems and 2) aren't engineering supremacists.

They are not *unrelated*, but they are different things.charity.wtf/2022/01/20/how…
Apr 11 12 tweets 3 min read
Say you want to modernize your org and introduce progressive deploys, feature flags, switch o11y vendors, etc. You could:

* roll each change out, one at a time
* change all at once, Big Bang style, migrating one service at a time

Has the Big Bang style EVER worked? For anyone? I can think of lots of examples of engineering orgs who *tried* the Big Bang style, but got wedged halfway through, or 20% of the way through.

I can think of lots of examples of orgs who are successfully bringing up *new* services on a new stack.
Mar 22 7 tweets 2 min read
Ooooohhh boy, this is a terrific question. I have written two closely related pieces,

* for engineers interviewing at a new company, on how to sniff out bad management culture:

* how to tell if the co is rotten on the inside: charity.wtf/2021/02/19/que…
charity.wtf/2022/01/29/how…
But both of those were written from the perspective of the engineer/interviewee, not the interviewer. The dynamic is different, for sure. 🤔

I would probably start by asking them why they became a manager, why they enjoy the job (if they do). (Softballs)
Feb 14 12 tweets 3 min read
Pro tip: any time you see someone confidently opining on what all good CTOs know or do, it is ✨bullshit✨

There is no stock template for CTO, or default set of expectations or responsibilities. It stands alone among the C-levels in that good ones are all over the freaking map. This may not hold true for publicly traded companies. But in my experience, a good CTO can be:

* over all of R&D
* over engineering, like a VP eng
* like a principal eng or architect
* team lead for special projects
* a great senior programmer

(continued... 👉)
Jan 22 7 tweets 2 min read
Yeah, this is a fair caveat. If you're already a decent senior engineer and manager, it's kind of possible to split your attention between managing a small team and writing code.

But you aren't going to improve at either skill set. Those cycles get devoured by context switching. Tech lead managers ("TLMs") are a mistake we make over and over in this industry. I've written about this a bit, but the definitive post was written by @Lethain.



Instead of being the best of both worlds, TLMs are poorly equipped to do either.lethain.com/tech-lead-mana…
Jan 8 12 tweets 3 min read
My coworker @suchwinston wrote a terrific piece on burnout before the break:

There's a reason why burnout and work/life balance are such evergreen topics, and it's not actually because the world is so terribly harsh and everyone is criminally overworked.honeycomb.io/blog/product-m… Just to be clear: some places *are* awful, and some people *are* criminally overworked. But burnout and work/life balance are an issue for everyone, not just those people.

I think this is bc there is no real "solution". Each of us has to find and maintain our own equilibrium.
Dec 29, 2023 6 tweets 2 min read
This is an astute point. For all the ink that has been spilled about what observability is or is not, or how generation 2.0 differs from 1.0, it's actually quite simple.

"Observability" was coined to denote the emergence of ✨high cardinality✨ support in telemetry and tooling. Cardinality, for those new to my feed 🤣 refers to the number of unique items in a set. Gender drop-down with three options? Low cardinality. Gender field you can write to? Much, much higher cardinality.

Metrics can't do high cardinality data. A metric can only be a number.
Dec 22, 2023 32 tweets 6 min read
About two months ago I wrote this thread about how we lost the battle to define ✨observability✨ -- to give it a real, specific, falsifiable technical definition, distinct from monitoring or telemetry.

I complained, I argued, I grieved...and now I'm over it. SO over it. 🙄 In fact, I've come all the way around: I AGREE.

It is less useful for observability to stand for a specific set of technical capabilities and outcomes (explorability, high cardinality etc).

It is more useful, and less confusing, to define observability as a property of systems.
Dec 2, 2023 6 tweets 2 min read
A couple months back this show called "Suits" was having a Moment, so I made it my falling-asleep show for a while.

As a show, it's ok. Pretty formulaic, but that should be *good* for winding down. It took me a while to untangle why it was getting under my skin so badly. Things that really bothered me about Suits:

* The people in charge refuse to tell anyone else what they are doing, or why they are doing it

* They rarely make requests of their underlings, and NEVER share context or background info; they bark orders and demand total obedience.
Dec 2, 2023 5 tweets 2 min read
This is a hard topic. ☺️ this statement in particular is one I might reject out of some mouths, but coming from @maggiepint, I know what she means. And yeah.

It is hard to hold a nuanced position on a complex topic, and harder still to convey. The dominant problem in tech is discomfort with/denial of feelings, so naturally the biggest backlash is about making it okay to be a human with physical and mental needs, normal emotional range, and family that needs attention.

But you can swing too far that way too.
Nov 27, 2023 7 tweets 2 min read
Yes. Being a great engineering manager who cares about people is exhausting.

Professionalism itself is a form of mild dehumanization; you and I set our identities and personal selves to the side, and interact with each other's professional functional self. We become cogs. 🙃 A manager needs enough emotional distance from their team to be able to make a decision because it is what's best for the system, not because it would make their team happiest.

You need to learn to use people like resources, but not lose touch with your humanity.. or theirs.
Oct 30, 2023 9 tweets 2 min read
So, we lost the battle to define observability. You know it, I know it. Observability was supposed to *mean* something, and in the early days, it did.

"Observability" once meant the kind of exploratory, open ended investigation our systems increasingly demand. And because you can't get it using old fashioned tools, "observability" also stood for all its prerequisites:

* high cardinality
* high dimensionality
* traceability
* structured log events
* rich context
* end to end instrumentation
* from the user's perspective -- each user's
Oct 27, 2023 8 tweets 2 min read
✨Happy Friday!✨ I'd like to take a sec to debunk a popular myth about the "deploy on Fridays" movement. Many seem to think we mean everyone should be deploying code right up til quitting time, every Friday.

This is not the point at all. This is NOT a great idea for most shops. I can see why people might be confused, because I would say that people shouldn't be *scared* of shipping on Friday. If there's some business or technical reason that occasionally keeps you writing code and shipping it all day Friday, nobody should run screaming for the hills.
Sep 26, 2023 8 tweets 2 min read
Over the last six months, SumoLogic, NewRelic, and Splunk have all been sold off to private equity firms (or noted tech graveyard Cisco).

I think it's harder than people realize to innovate for the next generation of observability when you were built to solve the last one. Observability is a UI/UX problem, an instrumentation problem, a scale problem... but it is also very much a data problem.

The shape of the data and the way that you store it will confine your product decisions for years to come, in ways that can be non obvious at the outset.
Aug 31, 2023 8 tweets 3 min read
This morning I recorded a new podcast with @theburningmonk, which was super fun. (coming soon!)

Yan pushed me on the question of Friday deploys. He says some people think it means that you should be able to ship every Friday at 4:59pm, then cruise on out the door. No?

Noooo. 🤦‍♀️ First of all: nobody should ever "deploy and walk out the door". Not on any day, not any time of any day. If you deploy your code at 10 am Tuesday and walk out the door, you're an asshole.

Your job writing code is not done until you've watched it run in production. Period.
Aug 27, 2023 45 tweets 21 min read
I just got back from @fintechdevcon in Austin. It was a super fun trip (notwithstanding the 105 degree air 🥵).

I gave a new talk while I was there, on why compliance and regulatory standards are ✨not✨ incompatible with modern development best practices. Image "Episode 01010010: Regulatory Compliance Is Not The Enemy

The fintech galaxy is ridden with cargo cults and myths about how regulatory standards prevent them from building software using modern best practices..."

(I was going to open with this vid, but ran out of time for A/V)
Aug 19, 2023 9 tweets 2 min read
Oh man, I also meant to ask... do people still *do* this? Run massive logging, indexing and searching clusters, I mean.

How do you justify the labor and opportunity costs, in an era where logs are a commodity and hosted services are dirt cheap? Serious question... I'd like to understand.
Jul 6, 2023 13 tweets 3 min read
Good people management has been very top of mind for me lately. 🤔 Honeycomb is big enough now that we are starting to develop our own "manager training" materials (except not just for managers, and more like an...immersive experiment?) There are so many big, hard, shitty lessons you have to learn that I don't think I've ever heard anyone talk about.

Or if they do, it's in the bloodless, studiously bland voice of the business aisle. "To be a good leader, you must build trust"?? NO SHIT Sherlock.