Thread by @QuinnyPig

The @awscloud explanation of their outage earlier this week has been posted.

aws.amazon.com/message/12721/

Because this is incredibly dense and technical, let me try to simplify it. I'm sure I will be condescendingly corrected if I get this wrong...

"We made a change internally that caused a bunch of internal things to become extremely chatty, like AWS employees defending the company if someone says something even slightly unflattering on Twitter."

"All of the chatty stuff made it really hard to understand what was going on because everything started behaving like CDK evangelists and never shutting the hell up for one goddamned second. As a result, engineers had to basically guess what was broken."

"It's always DNS, so they started there. It was not DNS, which is one for the record books. They then focused on moving traffic service by service, which eventually cleared the congestion issues."

"The thing that broke was basically 'the internal AWS network' which is kind of important as it turns out. A lot of things fell over as a result. We have a lot of learning to do about this newly discovered behavior and its triggers. We are deeply sorry."

As @lizthegrey points out, "making changes to DNS to mitigate" appears to be homed through us-east-1; using @awscloud for DNS looks like it may be A Mistake You Should Avoid as a result. This is important, concerning, and more than a smidgen disappointing as a customer.

https://twitter.com/lizthegrey/status/1469450928147234816

In all, it's a solid writeup that does an admirable job of transparency.