Thread by @Lethain

Over the last couple of years I've worked to build frameworks for infrastructure engineering teams to prioritize, invest and be durably healthy. This has culminated in four different approaches, one long blog post, and fifteen tweets.

lethain.com/how-to-invest-…

Aside: If this is a topic you're interested in, I'll be talking about this next month at @velocityconf on June 12th! (Glad to sign books if you've got em!)

conferences.oreilly.com/velocity/vl-ca…

Is infrastructure really so different as to need specialized approach to investment? Yeah, I think the average infrastructure team does have different constraints in two ways: (1) a lot more "forced" work driven by reliability, security, efficiency, etc and

(2) a much broader opportunity to invest today for more leverage tomorrow, because they operate so much of the underlying platforms that power a company's software.

Healthy engineering teams (product or infrastructure) mostly live in an innovation mode and pop into firefighting periodically, but it's more common for infrastructure teams to get stuck in firefighting for *years*. Also less common for inf teams to get practice in innovation.

Ok, back to four approaches to infrastructure investment!

Firefighting -- you can't fulfill the incoming short-term, forced work, typically due to stability or scaling pain. Do less any way possible, incl shed short-term work into the future. Focus on making *any* compounding progress. If you can't do less, you *must* hire.

Innovation (aka making discretionary investment) -- struggle to escape firefighting means that many infra engs have never gotten to do product mgmt-style work, and teams often struggle when they move out of firefighting. Priorities used to be very clear, not very unclear.

Top two tips for teams learning to innovate are (1) don't rush picking problem to solve, (2) don't rush picking solution to the problem you select.

Adopt the core of product mgmt: listen to user needs, prioritize problems by ROI, validate solution as cheaply/early as possible.

Principles -- many infrastructure teams get surprised by long-term, forced projects, like multi-year SOA or DB migrations. Identify properties that your team maintains (security, reliability, productivity, etc), set baselines for each, project 2 years out: what do you need to do?

Not identifying long-term, forced work early enough is the #1 that healthy infrastructure teams regress to unhealthy, firefighting teams. This is hard, because it requires seeing the future a bit. That's also why this is interesting work. (Insert Cassandra pun here.)

Unified -- finally, most organizations are composed of many teams, and each team is going to be in a different state. Some are firefighting, some are innovating, some are fluxing between states: you've gotta prioritize them all together somehow.

Prioritize 40% time onto users asks, 30% into platform quality (extended KTLO, esp including long-term forced work), and 30% into "key initiatives". Teams prioritize their own user asks and platform quality; key initiatives set by centralized org leadership.

This balances making decisions where information is best -- in the teams themselves -- and giving org leadership an adaptation budget for new priorities/emergencies. (Fixed costs of running infra are usually high, so 30% org-set budget is low but seems honest.)

How do y'all balance/prioritize work for your infrastructure teams/orgs?