Given that my daughter has broken her arm and Twitch / Amazon infosec have broken my heart, I distract myself from both with a thread devoted to answering this question, because I've lived it.
It's Day One, and you're the first DevOps hire at a startup. A bunch of developers probably interviewed you and almost passed on your hire because you suck at whiteboard algorithms; you saved it by whiteboarding AWS architecture on the fly.
What do you care about? Spoiler, it is absolutely not the bill. A bunch of developers have been running the infrastructure.
Be nice! They built something that's succeeded well enough to hire your ass; they don't deserve your scorn or mockery.
You're going to start with *not* the AWS account at all. You're going to log into GoDaddy or Namecheap or god alone knows what registrar hosts the business domain. It'll be under the founder's gmail address.
Transfer it to a solid registrar with a team approach; failing that, a group alias rather than a person.
There's probably one of them, which is bad. If it's older than four years old it's also suffering from the Underpants Problem: the root email address is the same as the founder gmail address linked to their commercial Amazon account they use to buy underpants.
You're going to want to untangle that, ideally via Control Tower. Start by moving dev because it's easier.
Rate limits are account-wide, so a dev script run wild can take down production via rate limit exhaustion.
Expect fun legacy things. People believing that there's a hard limit of 100 S3 buckets per account, so you might see "super-buckets" just completely full of nonsense.
Folks convinced that there's a 10 tag per resource limit instead of 50.
And then there's the psychology.
"Don't use service X, it's expensive" or "the dev environment is expensive" was true when it was a tiny company. "Expensive" is relative, and it's time to unpack those things. Pay money for value.
Understand the production app. Deploy a copy of it into a fresh AWS account, set up within your Control Tower AWS Organization. Watch all the assumptions break things!
That "mystery instance" or "big pile of S3 data" that no one understands or remembers? Use security groups / access policies to restrict all access to it for a while to make sure nothing breaks and people don't complain BEFORE you delete them.
What's worth remembering is that you are probably far more expensive than the savings you can inflict AWS bill if you're a small DevOps hire. You're there to empower the business, not explicitly shave pennies off the bill.
Put another way: if you're going to save $200K a year on the AWS bill, the company would have broken even by not hiring you, give or take.
That said you're still going to save money, because the problem is that you *do not know* what's in the account. That means there are aspects of the application you don't get, and there are almost certainly security concerns as a result.
Pretend you're going to be leaving in a month. Everything should be switched over to role accounts (devops@ instead of yourname@) for contact info on those apps that don't let you have multiple users.
Document the list of what you change as you go.
You should also probably grab a copy of the existing Terraform config and – haha, I kid I kid. Terraform that stuff.
Absolutely be helpful and friendly and not the department of no. Make your colleagues jobs easier; you're new, and they've been there a while. Become a blocker, add process instead of removing it? Good luck at your 90 day review.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
I often say that you should sponsor @LastWeekinAWS because you should. @davidcheal took me seriously and filled out the form!
I read his pitch, chatted with him a bit, declined to take his money, and have his blessing to deliver my feedback via Twitter thread. Let's begin.
First, I don't usually see the sponsor forms; it was passed to me. There's an editorial firewall!
Second, I won't take people's money if sponsorship won't help them.
David's product is thermite.red. Sponsoring me won't help him.
You're greeted by a mountain of text. People never read nearly as much as you'd think they would. The tagline ("meaningful insight into AWS usage") is half of the AWS partner network and a third of their own services if we're being honest.
One of the hands down most sobering conversations I’ve ever had was with a bunch of Very Savvy Investment Bankers about what exactly a total failure of us-east-1 would look like economically.
The *best case* outcomes closely resembled a global depression.
That’s comfortably within their probability models for “a US civil war.” They’ve drawn up maps that show likely sides for such an event and they plan accordingly.
These people get paid significantly more than most engineers do to consider risk.
This is why you’ll not find even the most die-hard all-in cloud customer who’s publicly traded who doesn’t have “rehydrate the business” level backups either on-prem or in another provider.
While I do enjoy Twitter, I believe it's important to "own my platform." As such, Twitter's not material to the functioning of my business.
But I do talk to a lot of folks here, and a "subscribe" button for @LastWeekinAWS in my profile can't hurt anything...
I started by emailing @revue and checking their Terms of Service. As of today, there's nothing against using their sign-up function and exporting the list to another platform unless I'm directly monetizing the subscribers via subscriptions.
And now, a thread of Ancient Sysadmin Wisdom: an incomplete list of things we have learned from decades of outages.
"It's always DNS." Yup. Everything relies upon DNS, those relationships are non-obvious, and some things like to cache well beyond your TTL.
"If an outage lasts more than ten minutes, it's likely to last for hours." Yup. Usually related to electric power, but this is a good rule of thumb for "do we activate our DR plan" decisions.
So today I'm going to store 1GB of data in @awscloud's S3 and serve it out to the internet. The storage charge is 2.3¢ per month the tier 1 regions.
Someone on the internet grabs that 1GB of data once. I'm paying 9¢ to send it to them. You read that right; just shy of four months' of storage charges to send it to the internet once.