Thread by @BinaryVixen899

Hi all, Alexandria Leal here, you may recognize me from films such as "Five Foxes to Watch out for" and "Witches to Burn ! 2017" but today (and largely this week) I'm actually going to be doing some work *Gasp* and livetweeting as much of #yow19 Brisbane as I can. :)

Would've gotten a headstart on it last week but it turns out that December is a super hard time for my PTSD and you're supposed to, vacation on this whole vacation thing so: spent last week doing both of those.

Anyway Dave Thomas is onscreen to introduce @/realGeneKim, I think I've already livetweeted this talk but going to do so again now that I'm in a better state to. (not tagging Gene because Twitter UI)

So Gene is talking about Unicorns and the effort that he was on to understand them over 20 years and how this brought him straight into the DevOps movement, and compares DevOps to LEAN principles which is a fascinating comparison and pretty apt come to think of it.

"The Phoenix Project" is the result of this 20 year journey.
Gene defines devops using the Jon Smart definition
"Better Value
Sooner. Safer. Happier."

So what is being tackled here? The Downward Spiral. Something that is dragging our organizations down aka technical debt. Picture the worst wire disaster you've ever seen here.

And this happens everytime we take shortcuts to get to market, manually configure something, etc. Technical debt gets worse over time. Data center fires at 5PM, weekend disasters. The dreaded Friday disaster. This all adds up and means developers have less time to develop

So we have this feeling that things are getting worse over time.
(BTW talk to your support people and principal engineers, they will tell you a lot about this.)

Problems
1.) absence of all the invisible structures needed to enable dev productivity
2.) orthogonal problem of getting data from where it is to where it needs to be
3.) strong opposition to new ways of working
4. ambiguity on what behaviors needed to support during a transform

Five ideals from "The Unicorn Project"
1. Locality and simplicity
2. focus flow and joy
3. improvement of daily work
4. psychological safety
5. customer focus

So, studies show that the business value of devops is even higher than we thought #YOW19

High performers /massively/ outperform their low performing peers. High performers have a 208x difference in deployment frequency, one hour and LESS of deployment lead time, time to restore is less than an hour, only 0-15% chance of Sev 1 outage or something else. #Yow19

That's stunning, especially if you come from a data center on prem environment.
So, this whole thing leads to 2x less time re-mediating security issues and 29% more time spent on new work. imagine what you could do with 29% more time on new work instead of firefighting!

This translates to profit, 2x more likely to exceed profitability goals if high performer, 2x more likely to hit other goals, retain employees, etc. This is the opposite of tech debt:

DevOps enables you to fully utilize all the miracles and promises of modern technology. We can safely quickly reliably securely achieve all the goals, dreams and aspirations of our business: look at the five ideals again

Ideal #1 Locality and simplicity:

So, Etsy Sprouter. Basically this is a story about engineers working together to creating value. In the bad old days, the devs would work on the front end, DBAs worked on the backend. Problem is for a feature to get shipped, /lots/ of coordination between these two groups

So they wanted to create something called SPROUTER that would enable these teams to work indepedently and then meet in the middle with sprouter.
However this led to a lot of outages, so they wanted to kill it.

And it turns out it worked, quality of deployments went up. Conway's law in practice

So here's the thing, you need to be able to have developers working independently, three teams to one team, you will see your lead time go down, etc. This is scary for a lot of people but it makes sense when you think of it, less org jumping = less lead time

So architecture enables teams to:
Make large scale changes without permission of someone outside the team or depending on other teams
deploy and release their service on demand
do their own testing on demand WITHOUT THE USE OF A SCARCE INTEGRATED TEST ENVIRONMENT

But, there's a lot of "ignore architects" stuff in tech, or there was. Because as it turns out your intro comp sci teacher was right and architecture impacts your daily work as a developer

First Ideal A Measure
Bus factor, how many people have to be hit by a bus to disrupt things
Lunch factor: How many people do we need to take out to lunch to get something done?
If it takes 200 people to do a complex deployment, that's not ideal, want a low lunch factor

Ideal: anyone can implement what they need by looking at one file, one module, and make the needed change

Not Ideal: everything has to be changed and the whole system needs to be understood

Ideal: can make and test our changes independently

Not Ideal: Integrated test environment (again, this couples you to every part of the system)

Ideal: EVERYONE can satisfy customer needs
not ideal: Teams have to escalate up down left right forwards backwards and inbetween

First Ideal: Data
So everyone has access to data we need
Not ideal:
Everyone has to wait months, quarters, hope the reports don't break

Second Ideal: Focus, Flow, Joy
Basically for Gene this helped him rediscover the joy of programming while being an ops person. #yow19

Ops vs Dev divide; Ops saves people from crappy dev code.
But, learning Clojure changed Gene's opinion on this. You can do so much with so little code these days!!! Also DevOps is a word for a reason :P
You can do so many things with so little effort these days!
#yow19

Functional programming is a good tool to think with:
core concepts:
Immutability
composability
Pioneered by LISP and ML
#yow19

We're seeing these concepts in other areas as well:
Docker, Kubernetes
You can compose stuff but if you want to change things you have to deploy. You can't rewrite history in Apache Kafka.
Immutability is really important and helps drive towards good practices. #yow19

Ideal: Functional programming allows us to focus on solving the business problem

Not Ideal: solving problems no one wants to solve
#yow19

"Things I detest now
Everything outside of my application
Connecting anything to anything
Bash
YAML
Patching"
XD #yow19

And it makes good sense in why this drives productivity and ultimately value. You get self-service, on-demand, etc. This leads to focus, flow, solving interesting problems, JOY

#yow19

"There's never been a better time for infrastructure and operations" @RealGeneKim #yow19

@RealGeneKim Flow: if you don't know what this is look it up. Basically, if you're doing something you love, if you're loving what you're doing you lose sense of time, sense of self. It just /flows/. Think of an otter drifting in a river. Seriously look up this concept. #yow19

@RealGeneKim Two types of learning:
Procedural, more in line with flow
One shot learning:
Solving something because you have to not because you want to. No long term value.
#YOW19

@RealGeneKim I think this also opens up doors, again with the idea of "lets' kill hero culture", a lot of stuff that fits into one shot learning is stuff we associate with the generic "Smart developer". But actually it's stuff that holds us all back

@RealGeneKim Measurement of success: starting the lead time clock at code commit. How long does it take to go from code committed to code in production?
#YOW19

@RealGeneKim (Gene if this is clogging up your mentions my apologies, twitter UI is terrible and wont' let me see if I'm @ ing anyone)

Ask yourself to what degree do you fear doing deployments? It says a lot about what's going on...
#yow19

So on the second ideal now that i've gotten twitter to stop spamming Gene's @ s.
Ideal: trunk based development
Not Ideal: "50 people and a war room" #yow19

If you can't merge your own changes with your own changes, you have a problem. #yow19

Third Ideal:
Improvement of Daily work

Not Ideal:
General Motors Fremont, had a terrible terrible track record because there were no procedures in place to detect problems or what to do when problems were found. HINT: That last one
#yow19

What does ideal look like?
Put as much feedback into your system as possible, because that's how you learn and that's how you innovate and fix more problems. #yow19

I love this story, this sticks through in my memory of this talk from the first time. Basically now, in this plant, if there's something wrong you pull a cord, if it took too long you pull it, if something is broken, you pull it. The entire line stops if you pull it: #yow19

The andon cord is pulled an average of 3,500 times a day in a Toyota plant, and at first that sounds bad, but then you look into why this happens. #yow19

Basically if you don't put in a fix here and now, you put technical debt in. You're also going to repeat this problem 55 seconds later. #yow19

Greatness isn't free:
The need to pay down technical debt
#yow19

So think of a fast push to market: Your goal is to get features! Whatever it takes
Features ->
Technical Debt ->
Defects ->
In the future:
Velocity <-
Defects ->
Everything is so hard and all the time is spent fixing defects
#YOW19

We've all had this. Like no matter where you work in tech this happens, and it makes people leave, it makes things less fun over time. People leave. And ultimately if you don't figure out how to cure this: it will kill your company #yow19

Risto Siilasma, NOKIA, Transforming Nokia:
Why did NOKIA axe Symbion? Because the lead time was so long that it was mirage. Yeah Windows Mobile didn't work out but the call was right. And even at the board level, detecting the presence of tech debt is CRITICAL. #yow19

And you see this over again:
Microsoft 2002: Bill gates memo about security
Google 2005: automated testing culture
Amazon 2004: Though shalt always go through APIs
Twitter 2008
it goes on
You have to find a way to manage technical debt or you will fail #yow19

And you don't necessarily have to bring it down to 0, but it does have to be brought down to manageable.

Allocate 20% of cycles to technical debt reduction.
Yes, 20%.
#YOW19

You have to make time for this. You /have/ to make time for this. If you don't pay the 20% tax you will pay a 100% task. #YOW19

so: enabling greatness
Ideal is 3-5% of devs dedicated to improving dev productivity

Not Ideal: "It's the interns, it's the low rate developers. These aren't important so give them to the mediocre people"
#YOW19

Great quote from Satya Nadella, who really gets this stuff I mean look at what he did with Microsoft, about how if a dev has to choose between a feature and dev productivity they should choose dev productivity #yow19

Not Ideal : no one cares if a build or test is broken
Ideal: fixing that becomes the most important work of the day
#yow19

Not ideal: Waiting days or weeks for a peer review, again we've all seen this
Ideal: if someone needs a peer review, drop what you're doing, do the review
#yow19

If you increase the number of and on pulls, the cycle time to get to production goes down......
So yeah, there's a lot to be discovered here
#yow19

Fourth Ideal: Psychological Safety
#yow19

DevOps principals and processes are being used in a wide variety of organizations with great success, there are case studies here on this.
#yow19

Your process sometimes outlasts the usefulness and you know, maybe you should kill some of that stuff. Good leaders should be looking at killing that stuff. That review board that modern processes replaced may be a nice weekly time waster, but it's also:
a time waster #yow19

So with psychological safety, there are several things you should look for:
Information being actively sought
Messengers are trained (as opposed to shot)
Responsibilities are shared
Bridging between teams &function groups are rewarded
Failure causes enquiry
New ideas welcome

This isn't just "Oh I wish we had that culture but that's not possible here", this is stuff that actually predicts performance

Project Aristotle found that yeah, this stuff makes great teams great.
Great practices enable:
blameless postmortems
chaos engineering
#yow19

Finally, customer focus; ideal #5

Core vs Context:
Core is what creates lasting durable business advantage
Context:
everything else
(Payroll, etc.)
The risk is that context starves core.
Bring your business to the basics, focus on what you do well and what creates value
#yow19

Not ideal: silo managers prioritize silo goals over business goals
ideal: it's all focused on customer value, if it doesn't create customer value why are we doing it at all?
#yow19

There's so much untapped value here. It isn't just FB, Amazon, Netflix, Google
There's tons of value to be created here, and this is possible at more than just those companies, it's possible at your company. Want to know more? Look for The Unicorn Project , follow @RealGeneKim ,