,
27 tweets,
6 min read
Read on Twitter
Hang on, I'm about to attempt one of the most challenging feats known to man: explaining *why* something that is obviously true to all who experience it, is in fact true. Namely:
"Why is it desirable for software engineers to practice ownership over their code?"
"Why is it desirable for software engineers to practice ownership over their code?"
First let's define software ownership. I've defined it in the past as,
* everyone with commit privs should also
* have the ability to deploy and roll back their code, and
* debug their code live in production.
Still more or less tracks, but now I feel these are less rules,
* everyone with commit privs should also
* have the ability to deploy and roll back their code, and
* debug their code live in production.
Still more or less tracks, but now I feel these are less rules,
.. more common manifestations of an attitude of ownership, or commitment to production excellence (as @lizthegrey puts it).
Another way to put it might be,
Another way to put it might be,
Your job is not done when you commit to master. Your job is not done until... well, until your code is decommissioned or you have handed it off to a new owner.
That's software ownership.
That's software ownership.
This usually means putting software engineers on call, since on call is (tragically) the primary way most teams support their code in prod.
(It definitely means that everyone who writes code is expected to support it in prod, in some way or another.)
(It definitely means that everyone who writes code is expected to support it in prod, in some way or another.)
The origin of this software ownership philosophy is not, as some insist, to torture developers. (Some of us may, but purely for recreational purposes!)
It's because we have learned through long hard suffering that ownership is the only way to deliver quality service for users.
It's because we have learned through long hard suffering that ownership is the only way to deliver quality service for users.
This becomes more and more true the more complex our systems get.
If you ship a change and lob it over the wall and it breaks, you are the person best equipped to debug and fix it. This truth sits at the heart of all the feedback loops we construct,
If you ship a change and lob it over the wall and it breaks, you are the person best equipped to debug and fix it. This truth sits at the heart of all the feedback loops we construct,
If you make other teams buffer between you and this truth, you are causing orders of magnitude of pain to be wastefully generated and smeared around on teams who lack your context and ability to alleviate the pain.
That's just cruel and stupid.
That's just cruel and stupid.
Oops gotta go.. ✨to be continued✨
I'm back! Resuming this thread on software ownership. (Sorry for the delay, twitter runs on nice -19.)
I don't remember exactly where I was, so let me throw out two links that should be read by anyone who claims to give a shit about the future of our profession.
I don't remember exactly where I was, so let me throw out two links that should be read by anyone who claims to give a shit about the future of our profession.
First, the Stripe developer report.
stripe.com/reports/develo…
Key finding: developers spend ~40% of their time on what I think of as "technical bullshit": deeply technical labor that does not move the business forward or provide value.
stripe.com/reports/develo…
Key finding: developers spend ~40% of their time on what I think of as "technical bullshit": deeply technical labor that does not move the business forward or provide value.
Second, the DORA report.
devops-research.com
Key finding: only 3.5% of teams are high performing on the only metric that really matters, which is time elapsed getting code to users. And velocity increases and failures decrease in tandem. (More: )
devops-research.com
Key finding: only 3.5% of teams are high performing on the only metric that really matters, which is time elapsed getting code to users. And velocity increases and failures decrease in tandem. (More: )
40% of software engineering cycles are all but wasted. FORTY. FUCKING. PERCENT.
It is a testament to human homeostasis that we accept this as somehow normal and okay. Our single most valuable asset, and we burn it up cause flames are pretty. (Then bitch on how we can't hire!)
It is a testament to human homeostasis that we accept this as somehow normal and okay. Our single most valuable asset, and we burn it up cause flames are pretty. (Then bitch on how we can't hire!)
But what lives in that forty fucking percent?
It's mostly the accumulated friction and debt accrued by years and years and years of building systems we don't understand and cannot observe. Patching symptoms and workarounds. Sedimentary layers of bugs submerged for eons.
It's mostly the accumulated friction and debt accrued by years and years and years of building systems we don't understand and cannot observe. Patching symptoms and workarounds. Sedimentary layers of bugs submerged for eons.
In the beginning, all of us wrote code on prod. When a user complained and caused us pain, we would telnet in and fix it. Problem solved. 👏
Shit got more complicated, we built tools and processes. Roles emerged and soon we had code experts (dev) and system experts (ops).
Shit got more complicated, we built tools and processes. Roles emerged and soon we had code experts (dev) and system experts (ops).
The emergence of software engineering as a profession was always quietly predicated on the existence of a human translation layer between systems and code.
Ops labor and toil (and 24x7 pager duty) made it possible for devs to treat systems like a abstraction. We buffered you from the consequences of your code.
It was killing us, but we took pride in it.
It was killing us, but we took pride in it.
We felt like heroes, which is super fun. We didn't realize that we had severed that swift, laser targeted feedback loop that leads to good software: you break something, you feel the pain of the broken thing.
The people who were causing the pain were not feeling any of it, not til days or weeks or months later.
The people who were feeling the pain had no context on what had changed or what the original intent was. They had no ability to fix it at the source either -- just duct tape.
The people who were feeling the pain had no context on what had changed or what the original intent was. They had no ability to fix it at the source either -- just duct tape.
This situation was shit. And thus devops happened.
🧚♀️ Wave one of devops: "ops must learn to write code!". Cool, done.
🧚♀️ Wave two: "devs must learn to build operable services." In progress, I'd say.
🧚♀️ Wave one of devops: "ops must learn to write code!". Cool, done.
🧚♀️ Wave two: "devs must learn to build operable services." In progress, I'd say.
But both of these are moralistic movements that sometimes get muddled about *why* they exist.
They exist to mend the feedback loops between engineers and their code, running on prod, by users.
They exist to mend the feedback loops between engineers and their code, running on prod, by users.
Meanwhile ops has been busy automating themselves out of a job. As we do. 💖🌈
I don't mean this to sound grim. It's not. There will always be good work for a good systems engineer, and the work is getting qualitatively less shitty and more fun.
I don't mean this to sound grim. It's not. There will always be good work for a good systems engineer, and the work is getting qualitatively less shitty and more fun.
The people who haven't quite dealt with reality are more software engineers than ops, in my experience.
Ops engineers are shipping code, embedding as SRE, and going to companies where they can solve operational category problems for everyone.
Ops engineers are shipping code, embedding as SRE, and going to companies where they can solve operational category problems for everyone.
Software engineers are just beginning to realize that their in house ops buffer for all the pain... is doomed. It's going away. You are going to need to own your shit.
If not now, soon. It's coming.
If not now, soon. It's coming.
I see extreme reactions to this every day on Twitter. And I feel you. You see the abuse ops has endured (and bragged about), and you're like HELL NO.
But that's not what's on offer. Abuse and misery are not inevitable when running highly available systems.
But that's not what's on offer. Abuse and misery are not inevitable when running highly available systems.
Abuse and misery pile up when you are building and running large software systems without understanding them, without good feedback loops.
Feedback loops are not a punishment. They mature you into a wise elder engineer. They give you agency, mastery, autonomy, direction.
Feedback loops are not a punishment. They mature you into a wise elder engineer. They give you agency, mastery, autonomy, direction.
And that is why software engineers, management, and ops engineers should all feel personally invested in empowering software engineers to own their own code in production.
FIN
FIN
