Thread by @mipsytipsy

there have been such hordes of journalists/analysts asking infinite versions of the same boring question for months now: "how is covid-19 going to change (infra|observability|tech companies|honeycomb)?"

-- that i am actually a touch peeved to have discovered a real answer. 🙃

which is this. across the board, companies are tightening up their internal efficiency, and this is accelerating their (the smart companies') adoption of observability.

when the future looks this uncertain, eng leaders become much more attuned to the cost of engineering cycles.

there are some enormous and heartbreaking human costs
being borne..layoffs, anxiety, etc 💔

but focus on internal efficiency? this is exciting. these are the times tech has made its great leaps forward into the automation age, when bets like "the cloud" were made and sealed.

anecdotally, we have seen a sharp decrease in the number of prospects who are considering using a vendor or building their own.

building their own was always a terrible idea. but as long as hiring more people felt effectively free, the opportunity cost was obscured from view.

once you sit with the fact that the engineering resources you have are the engineering resources you get for the foreseeable future -- the quarter? the half? the year? who knows? --

the cost of diverting a whole team away from your core business problems becomes suddenly acute.

i don't want to see anybody get laid off. 💔 but here's a fun experiment to do with your team, sort of like.. chaos engineering for humans:

"Could you 10x your business (revenue, requests, customers) with just the engineering team you have now?"

how? why, or why not? what would get cut? where would you break first?

do you know of any other companies doing roughly your volume, but with an order of magnitude fewer and/or greater engineers on their team? what are those teams doing differently?

your periodic reminder about the stripe developer report, and its findings that 42% of engineering work weeks are ... wasted. not moving the business forward, not innovating, just mired in technical bullshit.

stripe.com/files/reports/…

you cannot actually hire your way out of an overloaded engineering team. if you add more people to a team that is drowning, you will just have more drowning people before long.

you can only fix overload by reducing the workload to a tractable quantity. *then* hire more capacity.

you also cannot expect to hire engineers linearly with business growth, or you are screwed.

you need to figure out how to make every engineer dramatically more productive this year than they were last year, and next year more productive than this one. year after year after year.

if you are standing still you are losing ground. the bar gets raised every year.

so you need to steadily wring inefficiencies out of your systems -- make them easier to manage, easier to understand, easier to navigate and repro and maintain and upgrade, with fewer human cycles.

fortunately there are whole armies of tools and practices and communities out there to help you; but i feel like people get lost when they don't realize this fundamental truth of computing physics.

there's no such thing as standing still, you're actually sliding into entropy.

(i feel like that explains the lower 50% of the dora report 💔 they are bailing water out of the boat with all their might but not as fast as the water is coming in through the holes, so they never get the engine out of where it is submerged and cannot start up)

"internal efficiency" is one of those areas where we hyper-rational engineers don't realize that our guesstimates and intuitions are INCREDIBLY untrustworthy. 🙃

it takes a LOT to make us knuckle down and account for where our time goes. (like...global pandemic and depression.)

this is even more true without observability. you need some way of accounting for where the time is going, and how that changes over time, and playing with the data to find new efficiencies and black holes.

here's how to take a cudgel to your engineering team's inefficiencies:

instrument everything that happens between when you merge new code to master, and when it goes live. try using honeycomb free tier, instrument your pipeline as a trace...see @maplebed's talk infoq.com/presentations/…

once you see where the time goes .. you cannot. unsee. 😬

(this is the magic and the curse of observability-driven development. in the old days, you would stab around in the dark until you fixed a thing, then formalize it by adding a metric for some visibility. now, you shine a light on it, then you can't help but fix the whole mess.)