Thread by @QuinnyPig

Okay, okay. A thread on why this is a screaming red flag as a Cloud Economist...

https://twitter.com/817386/status/1588341919968481280

Most client requests are of the form "help us predict and control our spend." Occasionally it's "lower the bill." Almost never is it "lower the bill by X dollars."

That happens as a side-effect of the others; pursuing it as the defined goal is courting disaster.

There are optimizations that are always possible on a given an infrastructure. In Twitter's final report they show a $2 billion a year (ish) "cost of revenue," along with another $1.6 billion a year in "R&D."

Let's ignore that the majority of this is almost certainly people's paychecks and say that they're spending $2 billion a year on infrastructure.

"Cut it in half" is a very different thing at this scale than it is "your personal AWS account."

"A third of your bill is misconfigurations" doesn't scale, because someone notices! It's a variety of interconnected services, along with redundancy for systems, backups, etc. Very quickly you're not cutting fat, but muscle and bone.

It's also spread across Google Cloud, AWS, and on-premise data centers. At this level of public spend, there are multi-year contracts in place guaranteeing certain spend levels; an awful lot of this is amortized over multiple years.

You can save a boatload of money by, for example, deleting the data warehouse.

Yeah, that (probably) doesn't break Twitter's functionality, but suddenly your reporting doesn't exist anymore, along with your insights into what's happening on the platform.

A significant spend source is observability. Turning all of that off will cut a giant portion of your bill--but the next time Twitter goes down there's a decent chance that it never comes back up again.

You absolutely don't need backups until suddenly you very much do.

Further, I've spoken with a number of folks over the years who until recently worked at Twitter.

I'm not sure who needs to hear it, but these people aren't idiots. They're incredibly smart and capable. They weren't doing economically dumb things.

Let me summon the ghost of Parler here as well: they were booted off of AWS for *not having an effective abuse management process.*

Irrevocably break Twitter's content moderation, and the cloud providers will reduce your bill to $0 for you.

When someone tells me they want to cut the @awscloud bill, I always ask what they're trying to achieve.

There are ways to get to where they want to go without breaking the thing or making inane suggestions, generally.

"Cut $1 billion in costs" isn't the destination itself.

Twitter has also been around for ~15 years. A lot of infrastructure "cruft" has built up. "Is that important or can it be turned off" is the kind of thing best answered via asking people.

50% of Twitter was let go today, and your oral tradition just walked out the door too.

Cloud cost and cloud architecture are *THE SAME THING*. Finance can't solve this in isolation.

One of the best people I know for this kind of job found themself locked out of Twitter systems late last night.

As a dad, I'm disappointed that someone is not making good choices.

I have no particular insight into Twitter's infrastructure; this is all generalities.

Twitter is absolutely not on my Prospective Client list at The @DuckbillGroup, because we only work in an advisory capacity. I have seen nothing to suggest the new ownership takes advice.

This is exactly my point. As companies evolve they go through optimization passes.

Saving a billion a year without massive re-engineering might (**MIGHT!!!**) be possible if and only if they had done none of this work ever before.

They have.

https://twitter.com/10953972/status/1588621126036762624

Someone asked privately how I’d do a deeper pass. Okay.

It starts with building a model of unit economics. For every thousand daily active users (or whatever metric), there is a cost to service them.

Break that down. What are the various components?

Start with the large categories. Optimize there. “Half of it is real-time feed?! Okay, what can we do to reasonably lower that? Is a 30 second consistency model acceptable?”

“Huh, 5% is for some experiment we don’t care about. Let’s kill it,” becomes viable sometimes as well; this economic model lets the business reason about what’s otherwise an unknown cost.

I’m gratified by the number of current and former clients agreeing with this thread. ❤️