Thread by @lhochstein

Let's talk about reciprocity in software orgs, a thread:

According to @ddwoods2, "reciprocity" is one of the ingredients for achieving resilience. If one unit gets overloaded, other units will compensate. As he puts it: "I will help you when you're crunched".

Three strategies for dealing with a unit at risk of overload are:

1. Stand down (refuse additional work).
2. Shift work onto different units.
3. Provide additional resources to the unit under load.

As @this_hits_home notes, a useful analogy is thinking about an overloaded service in a microservice architecture. The strategies translate to:

1. Shed load (throttle!).
2. Redirect traffic to a different cluster.
3. Increase the size of the cluster (scale up!)

Now, let's consider a team within a software org as our unit coming under heavy load, and what we can do about it. I'm assuming a microservice architecture and you-build-it-you-run-it, since that's what I'm most familiar with.

The "stand down" strategy is the one I've seen most often used when a team gets really overloaded. The manager communicates to internal customers that feature requests are going to be delayed as the team recovers from going under water.

The other two strategies are harder to execute, because of the amount of context that an engineer requires to work effectively, which is why ramp-up times are long.

Since teams are generally specialized around a particular service, you can't simply move work from one team to another. You can't take a feature request for the "foo" service and implement it in the "bar" service instead.

Providing additional resources to an overloaded team is also hard. In the long term, you can staff up the team, but the real question is: "can you temporarily add resources until the positions are filled?"

In theory, you could have a reserve group of highly experienced engineers that were very familiar with the company tech, who could be dispatched to help out a new team, SRE-style. But I think that's a tough strategy to execute.

First, you need to staff up your reserves with experienced local engineers. That means you have to "poach" the talent from existing teams in your orgs, you can't hire from outside, they won't have the internal experience.

Then, even if you did manage to build a reserve force of experienced engineers, how effective would they be when deployed to help out temporarily on the underwater team?

There would still be ramp-up involved for these SRE reserves: even though they'd be experienced with company tech, they would lack prior experience with the specific service they're helping out on.

I'd love to see an experiment with the reserve approach, but I suspect it's too difficult to execute, and so "stand down" is the only real option. Fin