Charity Majors Profile picture
Feb 9 18 tweets 4 min read
a caviar-quality rant on deployment, security, testing ... actually more like five rants stuffed into a single trenchcoat. via @beajammingh

mumble.org.uk/blog/2022/02/0…
@beajammingh the title particularly caught my eye. for the past month or two i've been sitting on a rant about how i no longer associate the term "devops"** with modern problems, but with fighting the last war.

** infinitely malleable as it may be
yes, if you have massive software engineering teams and operations teams and they are all siloed off from each other, then you should be breaking down (i can't even say it, the phrase is so annoying) ... stuff.

but this is a temporary stage, right? a bridge to a better world.
...a world where you just don't build those stupid siloed towers in the first place.

a world where every engineer is writing and shipping code to production, and owning/operating the code they ship.
the reasonable distinctions to be made aren't between people who are writing code vs operating code, but between groups who are writing & operating different *kinds* of code.

sometimes that will be infra or platform teams, most commonly teams oriented around products.
if there's anything we can learn from the past 30 years of software trench warfare, it's that Bad Things Happen if you separate the roles of designing, writing, and maintaining software into three different groups.

and by that, i mean Bad Software Happens.
i don't say this to denigrate the immense amount of expertise that goes into operating large software systems, or running infrastructure at scale. my ops blood runs deep. i *know* how hard it is.
that is why it's so incredibly hard to find, hire, train, recruit and retain those people. which is just one of the many factors driving companies to outsource those skills to companies who can do it better than they can.
i've written about this at some length before. a few key points:

1) there are two types of software; the code you write and own yourselves, which defines your business, and all the code written by other people that you have to run in support of it.

thenewstack.io/the-future-of-…
2) "product code" is your crown jewel, it's what makes your company valuable. this is the code you should be writing, maintaining, and shipping as often as possible. you should know it intimately. you should instrument it intimately. you should practice observability and ODD.
3) "infrastructure" is everything you *have* to do, in order to get to the stuff you *want* to do. that doesn't mean it's not important -- doing infra well can be a huge competitive advantage. but it is inherently a thing to be minimized, and a candidate for outsourcing.
4) why? because engineering cycles are limited, and the more of them you can devote towards solving first-order business problems the better.

writing a calendaring library might technically move the business forward, but unless you're a calendaring company, you prolly shouldn't.
5) breaking up the software lifecycle into roles like dev, test, and ops inevitably produces such low quality software that devs refuse to be on call for it, so you have to hire armies of ops people as cannon fodder.

these teams, and there are many, need the devops gospel badly.
6) "devops engineer" seems to have become the term of art for "ops person who can automate things and write some code, but doesn't touch the crown jewels".

these jobs aren't going anywhere.. after all it turns out running software is hard, much harder than people realize. 🙃
7) but it's ultimately a kludge role, a term that muddies rather than clarifies any purpose.

ops folks are terrifically handy; if there's one thing we're good at, it's fixing literally anything. but if i was a devops engineer today, i'd be looking to the future & asking myself:
"what do i want to be building, five years from now?"

the era of reactive, bag-holding, firefighting ops work as a profession is coming to an end (and good riddance). in the future, everyone is a builder, an *engineer*, who then operates what they build.
instead of being a catchall role, devops engineers will be:

* build engineers
* internal tooling engineers
* infrastructure engineers who write and ship code at infrastructure companies
* SRE specialists who help software engineering teams with reliability and incidents
i guess what i'm saying is that in a post-devops world, operations skills won't be seen as a cost center, but as a core competency of every skilled engineer. and the specialty skill of a few.

that's all <3

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Charity Majors

Charity Majors Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @mipsytipsy

Feb 10
Several people asked this. It's a good question! I will share my thoughts, but I am certainly not religious about this. You should do what works for you and your teams and their workflows. 📈🥂☺️
1) "assuming you have good deduplication"... can a pretty big assumption. You never want to be in a situation where you spend more time tweaking dupe, retry, re-alert thresholds than fixing the problem.
2) having to remember to go futz with a ticket after every little thing feels like a lot of busywork. You've already committed some code, mentioned it in #ops or wherever, and now you have to go paste all that information into a task (or many tasks) too?
Read 12 tweets
Feb 9
I've done a lot of yowling about high cardinality -- what it is, why you can't have observability without it.

I haven't made nearly as much noise about ✨high dimensionality✨. Which is unfortunate, because it is every bit as fundamental to true observability. Let's fix this!
If you accept my definition of observability (the ability to understand any unknown system state just by asking questions from the outside; it's all about the unknown-unknowns) then you understand why o11y is built on building blocks of arbitrarily-wide structured data blobs.
If you want to brush up on any of this, here are some links on observability:

* honeycomb.io/blog/so-you-wa…
* thenewstack.io/observability-…
* charity.wtf/2020/03/03/obs…

and on wide events:

* charity.wtf/2019/02/05/log…
* kislayverma.com/programming/pu…
Read 16 tweets
Feb 6
Close! "If you're considering replacing $(working tool) with $(different tool for same function), don't do it unless you expect a 10x productivity improvement"

cvs to git? ✅
mysql to postgres? ❌
puppet to chef? ❌
redhat to ubuntu? ❌
The costs of ripping and replacing, training humans, updating references and docs, the overhead of managing two systems in the meantime, etc -- are so high that otherwise you are likely better off investing that time in making the existing solution work for you.
Of course, every situation is unique. And the interesting conversations are usually around where that 10x break-even point will be.

The big one of the past half-decade has been when to move from virtualization to containerization.
Read 12 tweets
Feb 3
Maybe not "full transparency", but I think *lots* of engineers chafe at the level of detail they have access to, and wish they were looped in to decision-making processes much earlier.
One of the most common reasons people become managers is they want to know EVERYTHING. They are tired of feeling left out, or like information is being kept from them (true or no).

All they want is to be "in the room where it happens", every time it happens.
I mean, that's why I got in to management. 🙃👋 And it works! It scratches the itch. Everything routes through you. It feels great...for you.

But you still have a team where people feel like they have to become managers in order to be included and heard.
Read 10 tweets
Jan 30
i'm only just seeing this now, but @copyconstruct knocks it out of the fucking park in this post. copyconstruct.medium.com/know-how-your-…

cindy and i have talked many times about the kind of blowback one gets for posting these kinds of things; writing it anyway takes guts, and nerves of steel.
@copyconstruct there's no shame in wanting power and influence to advance your agenda, if you're trying to fix things or improve complex situations. in fact, i think it's a moral imperative for people who care to not cede the space to those who ONLY want power and influence.
twitter is full of critics who have never written a line of production code or managed more than a pet rabbit. and that's fine.

but if you care about enacting real change more than being Right On The Internet, that means working through the vehicle of imperfect organizations.
Read 9 tweets
Jan 29
i completely agree. the more a company tends to talk about their diversity, transparency, etc, the more suspicious i get about how much they doth protest.

especially when they start conducting marketing campaigns around pay-to-play lists for "best employer" awards. 🙄
the best thing about real diversity (and real transparency) is that you don't have to THINK about it all the fucking time. it's not ✨broken✨ and in your face infuriating you with its brokenness all the time.
the most insidious thing about teams that aren't diverse is the constant cognitive and emotional load borne by those who happen to be different.

on a diverse team, people are relieved of most of that tax, and can just focus on being who they are doing what they do.
Read 4 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

:(