Thread by @geekgalgroks

It is appropriately cold today for Code Freeze.

Dead tree program?!

This is not the CodeFreeze in Finland.

This is the one at UMN and it is the 15th one.

What?!? I removed my shawl from my bag so I could bring my laptop but they have open spaces.

I can't suggest crafternoon!

Okay, it feels like the organizers are out of touch with technology conferences.

Feels like I am being lectured at.

The introduction for the conference is still going on.

Ahhhh!

Phones have been used to control pacemakers forever.

My grandfather had a handset that connected to their land line to send information back and reprogram his pacemaker with audio signals.

But now for the main event!

@mipsytipsy

She has been lost in the tunnels, sat in the water fountain, misplaced her slides, and had to search the airport for a warm coat because she didn't have one.

Observability and the Glorious Future

@mipsytipsy

Computers are hard.
We don't understand them.
But we ship things anyway.

@mipsytipsy

Track these four things to see where you stand:

How often do you deploy?
How long for deploys to go live?
How many deploys fail?
How long does it take recover?

@mipsytipsy

You don't need to be the best engineer to be on the best or elite teams.

Communication skills are super important!

@mipsytipsy

Good teams communicate with each other, care about their work, invest in incremental improvements, and are empowered to do their jobs.

@mipsytipsy

You don't know your system if you only look at it when it is broken.

@mipsytipsy

Happy customers and happy teams are the goal.

They are related.

@mipsytipsy

It is harder to predict how complex systems will break.

How do you watch for what you don't know will happen?

@mipsytipsy

This is the difference between known unknowns and unknown unknowns.

@mipsytipsy

Which is the difference between monitoring (known unknowns) and observability (unknown unknowns).

@mipsytipsy

Her slides are full of rainbows, ponies, and stars.

It is the best!

TLDR: Can you figure out where in the code to debug?

@mipsytipsy

Unknown unknowns may never happen again.

Which is why they are hard.

@mipsytipsy

Glass castles

Are fragile and forbidding to tamper with

@mipsytipsy

It should not be scary to fail!

Deployments are opportunities!

Think playground not glass castle.

@mipsytipsy

If your company shouldn't exist in the world...

Leave it!

Don't work on shitty projects.

@mipsytipsy

You can hit the piñata harder if you take off the blindfold first.

@mipsytipsy

On call should be shared by everyone who writes code and on call must not be miserable.

@mipsytipsy

Senior engineers need to AMPLIFY hidden costs!

Say it over and over.

@mipsytipsy

And that is it!

I will totally be grabbing unicorn stickers afterwards.

I gave Charity googly eyes because that is the way I roll.

Next speaker is Bonnie K. Holub with Observability in Big Analytics.

She has a PhD in Artificial Intelligence and has worked for many major companies.

AI has had many hype cycles.

Few early adopters remain.

Many struggle with it.

AI in this her examples are also known as data driven decisions.

You have to have use cases and be able to derive value from it.

The big picture

Callback to Charity's talk about what makes good teams.

There are many steps to digital transformation.

It isn't just slapping an algorithm on it.

Data is often in discrete and isolated silos which makes it hard to find the correct data.

Data scientists have to repeat their steps over and over again.

There are companies who retrain their models every night.

This is better than taking six months to build a model and nine months to deploy it.

If the model is 7% more predictive but costs 80% more to compute, takes three months to deploy, and is no better than random on live production data...

It isn't worth it!

Be smarter and more selective.

She just said pregnant people!

That is fricken awesome!

"Hockey stick up curve"

Yep, she is from Minnesota.

"Think of a sonogram machine in Baudette, MN."

Location of Baudette is basically Canada. It is right next to the little jut at the top of the state. en.wikipedia.org/wiki/Baudette,…

Ooh good question from the audience.

How can smaller companies compete or dominate their market? Aka how to convince customers give us data?

Have a path with customers to share data, convince them that sharing the data will help them too.

Lots of renegotiation will be required.

How will it help them to give you this data? What do they get out of it?

Next question: What do you think of intersection of data, analytics, and culture? Aka democratization of data.

She believes sharing data is how to build productive team.
Fail fast, learn, and move forward.

Culture eats strategy for breakfast.

That is it.

Okay, t-shirt of small, medium, large, extra large, and extra extra large isn't inclusive.

I am guessing they are straight cut too.

Yes it is the "prize" for open space topics but come on.

Next up is Mary Poppendieck with Fifty years of Observability

@mpoppendieck

She is giving us a quick history of 150 years of engineering because observability has been around for a long time.

A mail plane crash in Australia caused transition from canvas and wood.

The first jet plane crashed, then another crashed. It was determined that metal fatigue from square windows.

The son of the pilot from the mail plane crash created the flight data recorder.

Engineering: Creating elegant, efficient, safe systems.

We often forget about safety.

@mpoppendieck

[Gif: Galloping girdie (suspension bridge) moving in the wind]

Too much success can make you forget about failure.

Then success turns into a BIG failure!

@mpoppendieck

What is the software equivalent of metal fatigue?

@mpoppendieck

Operator fatigue may be the equivalent of metal fatigue.

@mpoppendieck

Incidents can turn into accidents and catastrophes.

@mpoppendieck

Tight coupling and complexity of the system can make incidents quickly change into accidents.

@mpoppendieck

And now it is a review of what happened at 3 Mile Island.

It was an incomprehensible system. Operators couldn't determine water level.

Aka blame the operator instead of admitting it is bad design.

@mpoppendieck

The three mile island reactor was barely controllable and incomprehensible.

@mpoppendieck

It isn't five decades of software engineering! We are getting one more decade!

@mpoppendieck

1967
She started her first job, in telephone network automation.

@mpoppendieck

The reliability goal for a subsystem was maximum of 2 hour of downtime in 40 years.

That is around 3 minutes only a year!

@mpoppendieck

Tolerance of failure

Because no failures is not an option.

@mpoppendieck

They didn't meet the 40 year goal.

Jan 15, 1990 the phone network was down for 9 hours.

@mpoppendieck

1970s ARPANET

Make hosts responsible for data exchange

Doesn't break down in a cascading manner.

@mpoppendieck

1970s PARC Inventions

Lots of we use today

Early tools for observability!

Xerox only commercialized the laser printer.

@mpoppendieck

1970s & 80s her job was engineering systems.

@mpoppendieck

In an plant that is potentially explosive, you don't want a single point of failure.

@mpoppendieck

Event-Driven
Object-Oriented
Edge computing

In her case it was machinery and pumps.

These ideas are not new!

@mpoppendieck

1990s

Relentlessly increasing complexity

Fault tolerant versus defect free

Internet versus heavy process

@mpoppendieck

2000s Internet scale

Google's chunk file system looks similar to her phone control system of the 60s

Fault Tolerance
Replication
Isolation

@mpoppendieck

Aka: Reinventing the old ways of reliability.

@mpoppendieck

2010s automated deployment, loose coupling, observability, machine learning.

Apparently 2012 was the big year for making AI possible.

Fault Tolerant
Fault handling: local responsibility
Observability: operator
Observability: historical data

@mpoppendieck

How do you know your automated deployment is safe and secure?
How do you mitigate the complexities introduced by microservices?
Does observability make our systems more reliable, resilient, and safe?
Does the learning sets used for AI introduce bias in our systems?

Does our software control systems with catastrophic potential?
Do we ensure the safety of the systems controlled by our software?
Are suitable feedback loops and forensic data used for learning?
Do we accept responsibility for the consequences of our systems?
@mpoppendieck

Think about safety with everything you write!

@mpoppendieck

Yay reminder to ask questions at the mic!

Complex interactions
Tight coupling
Catastrophic potential
Incomprehensibility

It was true of early train bridges and planes. It is true of software today.

@mpoppendieck

Question: Can you explain or go through a quick comparison of ethics of civil engineering versus software?

Can you ask what is the big picture? That is important.
What is this code for? What is it going to do?

Ethics is taught to civil engineers. When you sign your name on the drawing, you are responsible if anything bad happens.

Who is responsible for code?

@mpoppendieck

What is the role of regulation in software industry? Like transportation, is it feasible?

Software is in too many industries. We can't have one overarching regulating body.

Blanket regulations don't help. That is process. People always get around process.

We need to be thinking. What is the reason for writing it? What is safe in that world?

@mpoppendieck

There is a difference in engineering and IT.

Namely can you push back and ask the business what the spec is for.

@mpoppendieck

And that is it!

Time for lunch.

And they don't have a boxed lunch for me.

I got tacos from bar luchador instead.

Muwhahaha.

Anywho, I had a wonderful and lively conversation with @substars

He is wearing his awesome teal glasses as well.

But the next session is Observable Web Applications with Todd H Gardner

@toddhgardner

It is icebergs themed.

Angry icebergs

Client side is the top of the iceberg
Server side is the bottom

@toddhgardner

The users only see the top of the iceberg. They don't see or care about the bottom.

@toddhgardner

Users perceive the system only through the client side.

Front end devs end up triaging issues.

Server side issues show up as client side problems.

@toddhgardner

The client side is important to the customer
It is where front end triages
It is where server issues appear

Why are checks only on the server side? The last mile is the most important to the customer!

@toddhgardner

Todd is a full-iceberg developer

@toddhgardner

Ooh this is a brand new presentation!

He wants photos, tweets, and shares.

Muwhahaha

@toddhgardner

The client side is millions of distributed instances.

Each browser is an instance.

@toddhgardner

The browser is the client side runtime.

It renders the user interface.

And we attach network services to it.

And make it do interesting things with app code.

@toddhgardner

Anyone of these components can break.

@toddhgardner

"Software would be easier without users."

@toddhgardner

Browsers are all different.

Does your site work on a game boy? Do you care? Would you care if an important client still used it?

@toddhgardner

Error tracking

Recording unhandled exceptional states.

@toddhgardner

And now we are learning about a specialty ice website because it goes with the iceberg theme.

Did you know you can buy ice with logos on it?

@toddhgardner

Browsers don't have a unified single place to find errors.

404 are different from rejected promises.

@toddhgardner

"It is JavaScript. So we can redefine it to be our own thing!"

@toddhgardner

Wrap functions on a page so you can track errors.

@toddhgardner

Have an API for you to send interesting events and things to.

@toddhgardner

XMLHttpRequest() is a weirdly cased function.
XML is an acronym
But so is Http!

But that is the world of JavaScript.

@toddhgardner

I like how he is defining each term immediately after introducing its section.

@toddhgardner

Performance tracking is easier to get from browsers than errors.

@toddhgardner

Ooh this is a good point to bring up the "hadness" point. (Point where customers go from happy to sad)

If the service takes 100ms longer to resolve, will your customers care? This sudden change could be a problem.

@toddhgardner

Usage analytics isn't just for marketing and project management.

What are your users actually doing with your product?

@toddhgardner

Is your current system acceptable?!?!

@toddhgardner

Nice definition of funnels and why it matters to developers.

@toddhgardner

You can use usage patterns to determine if error or performance issues matter to your customers.

@toddhgardner

Limitation #0

You have to CARE about the client side.

@toddhgardner

Limitation #1

When there's no traffic

It can't be the only thing

Limitation #2

Too noisy

If you don't care about errors thrown by gameboys and smartfridges.

Limitation #3

Data PRIVACY!

@toddhgardner

Limitation #4

Networks and geography

@toddhgardner

Limitation #5

Non-ui services

If your clients don't hit it, you won't see it.

@toddhgardner

Limit #6

Deep trace Observability

Sometimes you have to go under the water.

@toddhgardner

Yay review slide!

@toddhgardner

Drink water!

Don't be a dehydrated noodle like me.

Final keynote

Casey Rosenthal
Evolving Chaos Engineering
@caseyrosenthal

I hope he got his lunch.

He and I were trying to find vegan lunches. I went and got tacos, he was going to order from Herbivorous Butcher. But that isn't part of the talk.

@caseyrosenthal

Going back in time to sailing and boat captains.

@caseyrosenthal

[Gif: Woman with a shiny red wig shaking her head and shouting "boats boats boats!"]

More risks meant more profit.
More risks also meant fewer deaths.

Because they have practiced!

They know what to do in storms.

@caseyrosenthal

Now a story about cobblers.

@caseyrosenthal

[Gif: Bowl of fruit cobbler with a scoop of ice cream.]

Oh, shoes.

But now it is about zookeepers and chalk.

@caseyrosenthal

Now fruit orchards.

@caseyrosenthal

And tables

@caseyrosenthal

Engineers work between economics, workload, amd safety.

@caseyrosenthal

Chaos engineering is the facilitation of experiments to uncover systemic weakness.

@caseyrosenthal

Reversibility is where software shines.

@caseyrosenthal

Now an story about a code freeze gone horribly wrong.

@caseyrosenthal

[Gif: Moss looking over at a fire burning nearby and ignoring it to look back at his computer.]

Software engineering is a bureaucracy.

Responsibility is taken away from the person doing the work.

It is the opposite model needed for knowledge workers.

@caseyrosenthal

So many great t-shirt slogans.

@caseyrosenthal

[Gif: Fry handing over a wad of cash while demanding "Shut up and take my money".]

Myth 1: Remove the people who cause accidents

Geometric proximity to a bad outcome doesn't mean you are guilty.

@caseyrosenthal

Myth 2: Document best practices and runbooks

Most of the interesting problems are unique. You won't always have a runbook.

@caseyrosenthal

Myth 3: Defend against prior root causes

There is no such thing as root cause.

Root cause isn't the same thing as least effort to remediate.

@caseyrosenthal

Rca pushes blame downwards.

It reinforces the hierarchy.

Myth 4: Enforce procedures

Myth 5: Avoid risk

Guardrails can prevent work.

@caseyrosenthal

Myth 6: Simplify

Not possible

@caseyrosenthal

Each property adds complexity.

Safer products are more complex.

@caseyrosenthal

Myth 7: Add redundancy

Sometimes you are just increasing risk of failure

@caseyrosenthal

Don't fight complexity.

Navigate it!

@caseyrosenthal

Testing is good but you have to know what to test for.

Experimentation means you don't know what will happen.

It is better.

@caseyrosenthal

"All roads are smooth on a flying merkel."

@caseyrosenthal

And the photo to go with the previous tweet.

@caseyrosenthal

And that is it!

Now it is open spaces.
I have no knitting with me so no crafternoon.

I'll probably do emails and whatnot instead.

[Gif: cricket on a floor with the words chirp chirp appearing around it.]

You need to be intentional about open spaces

This doesn't feel like that.

Hallway track is better.

Anywho thats it for code freeze this year.

Please remember if you have a happy hour, have mocktails! Or at least list your non alcoholic options.

Not everyone drinks.