Thread by @OrenNachman

A year ago I was lucky enough to be part of the launch of #AWS Incident Manager which shipped as part of AWS Systems Manager.

It was an absolutely wild rollercoaster ride, here's a smattering of thoughts...

🧵

https://twitter.com/awscloud/status/1391898772960124932

Amazon (and AWS by extension) is big on Ownership, one of the Amazon Leadership Principles - which means there is very little that is truly driven top down.

I led the team that built Incident Manager, but the real leaders are the team themselves - I was just there to help. 1/

The Incident Manager (IM) team was (and still is) comprised of multiple 2-pizza teams, each owning their own slice of IM end to end. This is an important concept - our devs own the product they are building, they own everything from planning through execution to ops. 2/

The role of the various managers on the team is to help unblock the team - we are there to make sure everyone can achieve their career goals, help with guidance and prioritization, and attend any and all meetings that devs don't want to be in! 3/

Importantly, devs are directly connected to our customers - from pre-beta to beta to GA, devs are involved in customer conversations. Multiple features were introduced or changed because a dev heard something from a customer that resonated with them and they brought it in. 4/

Launching products at AWS is massive undertaking. The bar for launch is super high and involves one of the few top-down driven areas at AWS. We followed a very well defined checklist to ensure that covers everything from Security to Regulatory to Availability. 5/

IM launched with some non-obvious features: 1: ssm-contacts stores contact details 2: multi-regional replication for consistency and availability 3: An extensive UX with multiple integrations (ChatBot, CloudWatch, EventBridge, SSM etc) in addition to a fully featured API.

6/

Launching all of this set us up for nearly every kind of review possible. It also highlighted the ownership that teams have at AWS - we got to work with Charlie Bell (SVP) and his incredible TA to justify schedule changes to add functionality that we deemed essential and to... 7/

... review some of that essential functionality. Having the opportunity to clearly and easily state "these are the customer benefits that are driving this new and unique functionality and here is why it is designed correctly" reinforced the combination of razor sharp customer 8/

... focus while still ensuring a high quality and deliverable bar. It also ends up being super empowering - you have a dev pitching their design to the Director of one of the largest services at AWS, explaining why it's the best way forward and impacting the entire roadmap. 9/

Launch day itself was an incredibly fun experience. A lot of the team are based in Sydney (yes, Australia), and even though it wasn't required part of the team opted to come into the office at an ungodly hour to drive the launch aligned to US hours. 10/

To be super clear - no one was forced to show up, it was in the spirit of being able to be together to drive the launch of the product we'd all worked on together.

COVID hadn't really landed in AU yet - we only had cases in quarantine and no requirements for masks etc. 11/

As soon as we launched we got the expected feedback about naming - AWS Systems Manager Incident Manager is a mouthful!

Naming at AWS goes through a long process that gets reviewed all the way up the stack. The naming for this one followed direct customer feedback... 12/

... primarily not wanting "yet another top level service" and wanting a descriptive service. Systems Manager was the obvious home for IM as the primary hub for operational tools at AWS.

The actual name, not to be confused with the role, is as descriptive as we could get! 13/

We still got the expected (and coveted 😄) tweet from @QuinnyPig...

https://twitter.com/QuinnyPig/status/1391940190067855360

14/

Launches like these are written up in our internal Change Management process. The majority of the launch is automated - push some config changes and components magically light up.

The minority includes some manual changes, mainly flipping switches on various websites... 15/

... to make the launch official or notifying partner teams (Marketing, Documentation, Support, SDK etc).

Our launch included 34 steps being run across multiple teams (I went a little nuts documenting things... image is cropped) and since I love emojis, ... 16/

... I used emojis to make it clear which team was responsible for which step. Blue steps are activities and Orange steps are validation steps. Each step has defined rollback procedures so that at any point we could yank the launch if needed. 17/

I'd love to say that the launch went off without a hitch - and for the most part it did, but we did have some polish issues that should have been caught before launch, but were missed. 18/

https://twitter.com/iann0036/status/1392027003612655617

While most of these issues were resolved within ~an hour of this tweet (some of them were deployments that were still in progress, and some required quick code changes) they shouldn't have happened.

It was a powerful learning moment ... 19/

... - no matter how many times you review your product, no matter how much you use it yourself, no matter how many (many, many, many) bug bashes you run, you will miss some things.

This doesn't mean you accept it - it means you work hard to find more mechanisms to catch... 20/

... these kinds of issues for future launches.

It also reinforced our connection to the community, I was super proud of my team who rallied around the launch, investigated these issues and fixed them on the spot. 21/

The IM launch will always be personally special, an amazing experience that started and finished with the incredible team that built IM.

It's easy to forget that large companies are still made up of small groups of passionate people who love what they are doing and are... 22/

committed to delivering incredible value to their customers. I continue to learn a ton from the team, and it's been an honour to be part of the journey from development to launch and continuing over the last year of incredible feature delivery. 23/

Massive Thank You to everyone involved - SDE, FEE, Product/Program Mgrs, Applied Scientists, UX, Docs, SDMs, partners and our incredible customers! (microfigs represent our customer personas, logos are teams 😊)

Fin 🧵

PS We're hiring (remote friendly) (US/AU)! DM me...