Thread by @marknca

yesterday I spun up 36x @awscloud EC2 instances to build out a weird sample data set

today, trying to get an exact cost for that work, it hits home (again) why @quinnypig has a very successful business

a story...

https://twitter.com/marknca/status/1425893353317621768?s=21

🧵 ☁️ #cloud #devops

ok, so I spun up the instances via python/boto3 (all old-school like) because I had a unique user-data script to each to execute and then shutdown

super simple 👇

🧵 ☁️ #cloud #devops

of course, the execution of this script took a while. about 6 hours. so I ran a couple quick, smaller scale tests and when I had things locked. I ran the script 👆

🧵 ☁️ #cloud #devops

36x:
1. new t4g.micro runs
- in default VPC
- with a role to write to an S3 bucket
- set to `terminate` when shutdown
2. it then installs a python module for the work
3. ...and starts working
4. uploading the results to S3

🧵 ☁️ #cloud #devops

this is an old (like me!), pre-container approach to quickly running a bunch of parallel computations

...but honestly, it might be the simplest way. and since EC2 bills per second, there's not a lot of waste

34 lines of python = 36 parallel processes

🧵 ☁️ #cloud #devops

these run for a few hours and now I want to know if my cost estimate ($3.00) was accurate. how much did this data set cost to create?

that SHOULD be easy to answer

<insert>@quinnypig laughing maniacally</insert>

🧵 ☁️ #cloud #devops

so, top right of the @awscloud console, Account Name 🔽 > My Billing Dashboard

$0.29 for Aug (it's a demo account with light usage 😉)

hmm, not quite right, let's click in...

🧵 ☁️ #cloud #devops

clicking that green bar takes me to AWS Cost Management > Home

now I have $0.65 in charges (up 124%)

on what? no idea the purple bar is just a 12-Aug charges grouped

🧵 ☁️ #cloud #devops

that purple bar takes me to AWS Cost Management > Cost Explorer and I have a new but familiar purple bar

THIS purple bar shows $0.60 in charges (down 7% from the last screen). still no details but I have options now! lots of clickable "Group by" links

🧵 ☁️ #cloud #devops

clicking "Service" groups the chart. in a questionable move, it shows me Secrets Manager (current charge $0.00) and some services I'm actually using here

cost still $0.60 but the chart isn't useful for single expenses . handier for trends over time

🧵 ☁️ #cloud #devops

what is handy is the chart further down. it shows a detailed service breakdown

for my data set run:
- Config $0.51
- S3 $0.01
- EC2-Other $0.06
- EC2-Instances $0.02

um....wtf?!?

🧵 ☁️ #cloud #devops

a quick side trip to AWS Config, and yes, recording is on which explains the $0.51 charge.

as I said, this is a demo account and I had Config enable previous but wasn't being charged because nothing was happening for a while

turned that off for now...

🧵 ☁️ #cloud #devops

so how much did I actually spend on EC2? well that'll depend on how long each of those instances ran for. I'll check in the EC2 console, because apparently this isn't a billing question?!?

🧵 ☁️ #cloud #devops

there are no more instances running, so a filter _might_ be able to display then. sadly, there's no default for "stuff from yesterday". I try launch time...

🧵 ☁️ #cloud #devops

I get cheeky and try again with the exact timestamp of one of the RunInstances calls...

🧵 ☁️ #cloud #devops

ok, no dice in the EC2 console. let's check for TerminateInstances in events in CloudTrail

docs.aws.amazon.com/AWSEC2/latest/…

🧵 ☁️ #cloud #devops

um?

so there's only one call to TerminateInstances from the time period and that's from when I terminated a test instances in the console.

oh snap, TerminateInstances doesn't fire when the instance self-terminates via a shutdown command 😬

🧵 ☁️ #cloud #devops

ok, so cost explorer doesn't have the data. EC2 doesn't have it. CloudTrail doesn't have an end timestamp via TerminateInstances. the bill hasn't been generated so I can't download the CSV

hmmm...

🧵 ☁️ #cloud #devops

oh, ridiculous but it works: I have the start time of each instance via CloudTrail > RunInstances. I also have a termination timestamp through S3!

the last thing the instances do before shutting down is copy over their results

🧵 ☁️ #cloud #devops

after some copy & pasting, that gave me a rough set of timestamps to work with. some regex & Excel magic from there and I got an average execution time of 0.425 of a day or 612 minutes

🧵 ☁️ #cloud #devops

the t4g.micro were billed at $0.0084/hr or $0.00014/min

that gives us an average per instance cost $0.08568

for a total of $3.084! (my estimate was $3.00)

🧵 ☁️ #cloud #devops

the exclamation is there because I was happy to get the end of the this 🤬ing thread. it should not take this amount of work to figure out the actual cost of an experiment after you run it

🧵 ☁️ #cloud #devops

if I wait a few days, the Cost Explorer should have the data but why do I have to wait?

the good news? the EC2 charges line up with what I was expecting

of course, there's some additional billing for EBS usage during the run, S3 storage, but no worries

🧵 ☁️ #cloud #devops

takeaway from this story: @awscloud billing i s mess. a complete & utter mess

as a builder, make sure you forecast costs ahead of time & circle back to make sure they line up

the @awscloud Well-Architected Framework can help you build this muscle...

🧵 ☁️ #cloud #devops

as an organization, keep hitting up @awscloud to make this better. it's a top #awswishlist item for a lot of ppl, someday it'll get better?

...and remember @quinnypig is worth 10x what he charges: duckbillgroup.com/contact/

/🧵 ☁️ #cloud #devops