Thread by @QuinnyPig

So a kind member of the audience gave me access to their @awscloud bill for livetweeting purposes. I tore it apart live on Twitch last week, but now comes the tweet thread.

The bill's from last month but uses the @awscloud logo from yesteryear for no apparent reason.

The bill cost $2K in actual cash, and another $31K in applied credits. People generally start caring a whole lot more about the bill once the credits run out.

People LOVE to go into these things alphabetically, so the 63 requests to an API gateway will get scrutiny despite costing quite literally zero dollars.

We also see three Budgets are configured. Good choice!

CloudFront goes on and on, but most of the spend comes from Australia, and the data transfer attendant.

I'm still annoyed that HTTPS costs more than HTTP; you don't charge extra for security.

Unless you're using CloudTrail for data events, it should cost nothing; the first trail for management events is free, it costs past that.

Not that I care much about 36¢, but... hygiene.

It looks like Singapore and Sydney are where the action is happening.

At some point the CloudWatch bill begins to really, really hurt.

70¢ per gig of logs ingested is well into the realm of "pay something a lot more user friendly" pricing.

3 cents cheaper in Sydney than Singapore, but that doesn't really help.

Data transfer. Not the end of the world here, as a percentage of the bill.

This tells us something about their architecture. ~500GB transferred out, ~620GB transferred in, and ~10TB transferred between AZs (it gets billed twice).

Why are they taking data in, then moving it back and forth this much? Kafka, Cassandra, or Kubernetes is likely.

Device Farm is being used likely, and for once as something other than winning a trivia question. It's a weird service.

I'd want to break out my analyzer here and figure out whether there's an on-demand billing model for these tables. Alternately, buying committed capacity might drop this too. It doesn't appear from the round numbers that anything is autoscaling.

And despite all the lies we tell each other about serverless, higher level services, etc...

The big money is always in EC2.

A thousand bucks a month for managed NAT gateways (6 of them, indicating multiple accounts). That's a fair bit of money that turns into basically $0 if you run your own NAT instances instead.

Their EC2 usage is of latest-gen instances by and large. This doesn't feel like an older account. Hmm.

There's no spot usage. That might be an opportunity for some workloads.

EBS! I'd urge looking into gp3, determining if io1 is needed at all (rarely! It's usually an instance throughput limit!), and checking what the deal is with half a terabyte of snapshots.

"Unattached EIPs cost money. We should turn those off. I am very smart." EIP usage is almost always a rounding error. It's... not the first place I look for savings.

Found the bastard! EKS without Fargate costs indicates that they're running Kubernetes. That'd speak to the data transfer cross-AZ charges. Does the newest version respect topology yet? This one doesn't.

Nobody's figured out how to get ALBs in place. This feels more and more like a lift/shift story.

Redis has its adherents and detractors. Here, we see one of its customers.

KMS indicates multiple keys are in use. Good! Don't roll your own!

There are many different queues you can choose to use. AmazonMQ is the worst of them. Nobody likes RabbitMQ / ActiveMQ if they have other options.

Amazon Polly got taken for a test drive too. That's... not most environments.

Almost skipped Lambda! It's still well within free tier limits. This usually means it's still in testing, or is being used only for a few gaps between AWS services / replacing a cron job somewhere.

The miscellaneous IO charges for Aurora are the biggest barrier to adoption. There's no good way to predict them without testing a workload to see how it behaves. Ugh.

Fortunately they also use a better database.

No comment on Secrets Manager. The 1 year purchase on Savings Plans indicates they're likely planning to move some stuff around. The no-upfront means they can pay credits for this.

Something internal emails out notifications or similar. This isn't heavy usage at all.

SNS is all within free tier. Shoutout to @awscloud for listing "number of requests" to three decimal places of precision. Nailing it.

Oof. Almost no S3 usage. That tells me their applications think in terms of block storage, not object. This has consequences--financial, durability-wise, and how "cloud native" it'll be on someone's scorecard.

More requests than storage, which is an interesting pattern.

Definitely multiple accounts in here, which is why we see different support tiers.
And this rounds out the bill.

So analysis!

Lift-and-shift style migration, hasn't quite shifted into full on cloud native. Multiple accounts, which means dev and prod are distinct (I hope). At this scale there really aren't too many "expensive mistakes." They'd be best served by building further.

When those Activate credits expire someone's going to get an alarming (to them) bill, but they're not even close to "get an enterprise discount in return for committed spend" agreement level; that's ~$1 million a year.

This exploded bill view tells us a lot, but doesn't get into the weeds enough to say whether Spot would make sense, nor if they're on the right instances for the workload. Those conversations are generally highly nuanced.

So there you have it.

I do want to point out that @jesse_derose, @nerdypaws, and @petecheslock don't usually even look at this view; all three of them are better at bill analysis and reduction than I am.

But that was fun. Thanks to the anonymous donor for sharing their bill with us.

I'd be willing to make this a recurring post topic if folks enjoy it!

And of course, if you're beset by large AWS bills that vex you, we at duckbillgroup.com would be thrilled to chat with you.