Thread by @QuinnyPig

Well this got my attention. @nathankpeck was kind enough to let me into his changelogs.md account with my standard bill analysis IAM policy, so I tore this one apart for fun.

I don't just shitpost on Twitter I also fix horrifying @awscloud bills.

It's threading time!

https://twitter.com/nathankpeck/status/1232793680756514818

I have this ridiculous notion that bills should be tackled big-numbers-first. I don't care about Alexa for Business when it costs $3, but everyone who reads the bill alphabetically sure does. To that end, where are the spendy bits?

In a "real" cost engagement I'd be working on the CURs directly with a pile of custom tooling, but for this I'll use Cost Explorer. I give it crap and my dreams are big, but it's become perfectly serviceable. In years past I'd have had to use a third party tool instead.

So changelogs.md cost $115 last month. EC2-other is the big category. A bit of dive into there and it's our old friend, Managed NAT Gateway.

This charges per hour, and 4.5¢ per GB sent through it. Use it to store data in S3, you're paying a data processing fee of 2x the monthly storage cost of that data. Here the data transfer isn't bad; last month the data processing fee was $4.60. The instance charge was $33.48

We also have the code and the architecture diagram. Yay transparency! Shove ElasticCache and its attendant Lambda functions into public subnets and that knocks $38.08 off the bill. @nathankpeck gets a JIRA ticket there, and we move on.

@nathankpeck Next up is a single ALB. The ALB instance cost $16.74 last month, and removing it is architecturally painful, so we let it be. LCU-Hours, the least intuitive unit known to humankind, cost one penny in January.

Now we come to Elastic Container Service, the dark domain in which @mndoci dwells. Cost Explorer of course calls it "EC2 Container Service" because even the brand folks in @awscloud marketing dare not tread in the land of the AWS Bill.

@mndoci @awscloud Back when this was built, Fargate was expensive. It saw a 40% price cut, then had Savings Plans, then had a Spot option.

The fun part about Spot is that it saves money, but it's impossible to predict how much. This makes it a non-starter in some finance-first companies.

So @nathankpeck can deterministicly save $8.26 a month with a Savings Plan, or "more than that, but how much more varies" with Spot.

@nathankpeck Next we come to S3. This is fascinating; 22¢ in storage, $13.47 in request charges. It'd boost the DynamoDB a bit, but looking at the architecture keeping a hash of each object stored in DynamoDB would reduce a lot of duplicate requests. By a lot!

DynamoDB is fascinating; @nathankpeck just changed it over to OnDemand. We'll see what that does over the next month, but the same workload hits provisioned break-even at 14%; ondemand is over 7x more expensive than Provisioned, but you've gotta know your workload.

@nathankpeck Next we see $12.65 in ElastiCache; a single node. Make sure it's the right size, buy an RI, and spend $5.95 instead.

@nathankpeck We're now in the realm of the piddly. 74¢ CloudWatch, 40¢ Secrets Manager (free if you use Systems Manager Parameter Store). The juice here isn't worth the squeeze, at it were.

@nathankpeck So at minimum we've found $53.04 in mostly-easy savings on a $115.06 bill, or 46%.

Now here's the messed up part: @nathankpeck works at AWS and is incredibly intelligent. *NOBODY* has all of this in their head! In other words, maybe you shouldn't beat yourself up about your AWS bill.

@nathankpeck And yes, all of this is small-dollar stuff--but if this were to be scaled out, the numbers expand rapidly. Conversely, you probably don't want expensive engineers spending weeks to save $200 a month.

If this resonates with a problem you’re having, check out duckbillgroup.com and let’s chat.