Thread by @lhochstein

Why incidents are a good opportunity for learning, a thread.

First, let's talk briefly about *what* we want to learn. One valuable topic to learn about in an organization is how the organization actually works, in the what-do-people-do-day-to-day-to-keep-things-running sense.

(I'll leave why this is valuable to a future thread).

It may seem apparent what normal work looks like, but it's almost entirely hidden. Our work is mostly invisible. How much stuff do you do that your manager doesn't see? How about ICs on other teams? I bet there's stuff you do that your teammates don't even know about.

In principle, there are many different ways you could learn about how normal work happens in an organization. You could sit in on meetings, hang out in Slack channels, join mailing lists, read memos, look at source code, and so on.

In practice, just randomly sampling from those data sources isn't a useful way to learn, because it's unfocused. If you randomly sit in on meetings, read Google docs and look at internal pull requests, you're going to get bored pretty quickly. Just ask a new employee.

An incident, on the other hand is a mystery. It's an operational surprise, something that nobody expected to happen. And humans seem to be wired to be interested in mysteries.

The surprise becomes a lens to focus your attention on specific work in the org. How did the system get into a state that we never expected it to be in? There are many things that had to happen for the system to get into that state. What were they?

As you explore the different factors that all had to be true, and you talk to people that were involved, you will learn things about how they do their work. That's assuming you know how to ask the right sort of questions.

You'll learn about the script that was written as a one-off to address some data inconsistency. "Oh, what was the source of that data inconsistency in the first place?" you might ask. And then, off you go solving another mystery.

None of these explorations need be related to the impact of the incident, or preventing a future, similar incident.

One of the most valuable things to learn is how people do their work effectively, often in spite of constraints in their environment. The fact that our jumping-off point was about something that went "wrong" is an unfortunate irony, since people want to focus on prevention.

It just so happens that it's difficult to study "how things go right" within an organization without a focus. On your own team, you'll learn by working alongside your teammates, but that restricts the learning to your local team.

And so, it's useful to think of operational surprises as an opportunity to leverage the mystery-loving aspect of the human psyche in order to learn more about how work gets done in an organization. Fin.

(Bonus tweet): If doing this kind of work interests you, we're hiring for it on my team: jobs.netflix.com/jobs/869465