Profile picture
John Allspaw
@allspaw
, 3 tweets, 2 min read Read on Twitter
After @ph and I gave a talk about dev and ops cooperation (and a bit about “continuous deployment”) at @velocityconf in 2009, I was told by some attendees that:

1. We were “fucking lunatics”
2. Deserved to be fired for suggesting the things we spoke about were possible
Many believed them to be radical ideas at the time, some still do!

Similar reactions came about the “blameless postmortems” blog post I wrote in 2012.

Some concepts and ideas take time to grok/work through/spread/refine/evolve, even those that evoke strong defensive reactions.
I believe the same is happening wrt concepts of Resilience Engineering (and therefore progressive incident analysis expertise) right now.

Very happy to have fellow travelers on this front, just like I was very happy to have @ph ten years ago!
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to John Allspaw
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

More from @allspaw see all

Profile picture
John Allspaw
@allspaw
"Resilience engineering is about identifying and then enhancing the positive capabilities of people and organizations that allow them to adapt effectively and safely under varying circumstances. Resilience is not about reducing negatives (incidents, errors, violations)."
1/8
"Resilience engineering is based on the premise that we are not custodians of already safe systems. Complex systems do not allow us to draw up all the rules by which they run, and not all scenarios they can get into are foreseeable."
2/8
"Systems are not inherently safe: they need to operate under competitive pressure, having to meet multiple conflicting goals at the same time, and always with limited resources. People and organizations have to create safety under these dynamic circumstances."
3/8
Read 8 tweets
Profile picture
John Allspaw
@allspaw
The tendency to view incidents as having neat boundaries between "phases" of them is problematic at best. A good deal of traditional thinking about learning from incidents depend on this construction...
But this isn't what we find when we look closely at *real* incidents in the wild. They tend to be much "messier" than the above picture purports.
What we find more often are multiple and parallel lines of activity taken by multiple people who can sometimes have contrasting views on not only a) what's happening, but also b) what that potentially means for it getting worse or better, and c) what options are available to do
Read 6 tweets
Profile picture
John Allspaw
@allspaw
I am somehow surprised when I encounter folks with such strong beliefs in the "Humans-Are-Better-At/Machines-Are-Better-At" approach to designing software. At this point I shouldn't be surprised, but still am. 1/n
This "HABA-MABA" philosophy has been so ingrained that it almost flies under the radar as worthy of attention to some.

Who will argue with "make the computers do the easy/tedious stuff so humans can do the difficult/interesting stuff"? (apparently, I will) 2/n
This notion was first described in the late 1940s, and it was known as "Fitts' List" - original is shown here... 3/n
Read 12 tweets

Related threads

Profile picture
Liz Fong-Jones (方禮真)
@lizthegrey
I'll be livetweeting @mipsytipsy's talk on testing in production in a few minutes... #VelocityConf
@mipsytipsy The room is full, and we found her! [ed: hi, I was marshalling her from her hotel! this place is a twisty maze!]

People feel trepidation or like they're yoloing like a cowboy when they hear "testing in production". It gets peoples' attention. But it has a bad rap. #VelocityConf
The reality is that we all do it all the time. Responsible engineers test in prod, irresponsible engineers don't.

It's actually a false dichotomy; it's not a binary. You can test in production *and* before you reach production. #VelocityConf
Read 34 tweets
Profile picture
Liz Fong-Jones (方禮真)
@lizthegrey
.@gwenshap on running out of disk space: you can add more disk, or turn it off, change retention settings, and then back on.

the worst thing you can do is truncate the disk log with /dev/null while Kafka is running. Corruption!! #VelocityConf
@gwenshap When you're in a hole, first stop digging. If you don't know where the holes are, stop digging.

If you over-provision, upgrade to pick up bugfixes, feel comfortable asking for support, and don't "just tweak the parameters because"... you'll be better off. #VelocityConf
Subtle edge cases: if your disks are full but you discover it through an asynchronous flush... and it then crashes... and there aren't enough caught up leaders for quorum... data loss.

a human needs to decide what the new golden state should be after data loss. #VelocityConf
Read 18 tweets
Profile picture
Liz Fong-Jones (方禮真)
@lizthegrey
The final keynote of the morning is my former colleague @aknin, on how to be an SRE (and no, it's not starting a new org structure)! [ed: I've reviewed this talk in advance and it is *killer*] #VelocityConf
@aknin He works with teams to understand what reliability means, and then to quantify that definition.

This requires SRE skills, development skills, and product/project management skills. #VelocityConf
We should first start by defining SRE. Not all of us have the same definition!

Engineering -> using scientific principles to design/build things.

But how do you apply science and design to reliability? #VelocityConf
Read 19 tweets
Profile picture
Liz Fong-Jones
@lizthegrey
Final talk I'll be getting to at #VelocityConf before I dash to Toronto: @IanColdwater on improving container security on k8s.
@IanColdwater She focuses on hardening her employer's cloud container infrastructure, including doing work on k8s.

She also was an ethical hacker before she went into DevOps and DevSecOps. #VelocityConf
She travels around doing competitive hacking with CTFs. It's important to think like an attacker rather than assuming good intents and nice user personas that use our features in the way the devs intended things to be used. #VelocityConf
Read 36 tweets
Profile picture
Liz Fong-Jones
@lizthegrey
Leading off the k8s track today is @krisnova on migrating monoliths to k8s! #VelocityConf
@krisnova [ed: p.s. her ponies and rainbows dress is A+++]

She starts by providing a resources link: j.hept.io/velocity-nyc-2…

The problems we're solving:
(1) why are monoliths harder to migrate?
(2) Should you?
(3) How do I start?
(4) Best practices #VelocityConf
.@krisnova is a Gaypher (gay gopher), is a k8s maintainer, and is involved in two k8s SIGs (cluster lifecycle & aws, but she likes all the clouds. depending upon the day). And she did SRE before becoming a Dev Advocate! #VelocityConf
Read 29 tweets
Profile picture
Liz Fong-Jones at #GoogleCloudSummit Toronto
@lizthegrey
I'm excited to see @alicegoldfuss's talk on container operations for the first time! #VelocityConf
@alicegoldfuss She's telling us about the idyllic picture painted by advertisements (for instance, for cars) with the fancy 0-60 numbers... vs. the tiny small words "laboratory setting" or "test track". #VelocityConf
That's how @alicegoldfuss feels about container talks. Presenter comes out and says "BAM, I will revolutionize your infrastructure? What's that? Run more than one container? No problem! [with shipping container images overlaid on cars]" #VelocityConf
Read 29 tweets

Trending hashtags

#war #Iran #VelocityConf #velocityconf #GulfOfOman #Velcoityconf #USA #JCPOA #CIA #architecture #DoubleStandards #US #UnitedStates #EU #nanotechnology #Saudi #PersianGulf #WheresManafort #NoWarWithIran #Liars #OilTanker #MuellerReport #softwarearchitecture #IranSanctions #bullying
Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!