Thread by @HonkHase

Facebook engineering: More details about the October 4 outage

Let's have a look at it 👌 /1
engineering.fb.com/2021/10/05/net…

Cool, they do crisis trainings!

"Helpfully, this is an event we’re well prepared for thanks to the “storm” drills we’ve been running for a long time now." /2

"In a storm exercise, we simulate a major system failure by taking a service, data center, or entire region offline, stress testing all the infrastructure and software involved." /3

Feedback loops, cool cool 👍

"Every failure like this is an opportunity to learn and get better, and there’s plenty for us to learn from this one. After every issue, small and large, we do an extensive review process to understand how we can make our systems more resilient." /4

But what did really happen? 🤔

Here we go... /5

"And in the extensive day-to-day work of maintaining this infrastructure, our engineers often need to take part of the backbone offline for maintenance — perhaps repairing a fiber line, adding more capacity, or updating the software on the router itself." /6

"This was the source of yesterday’s outage. During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network..." /7

"effectively disconnecting Facebook data centers globally. Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool didn’t properly stop the command." /8

"This change caused a complete disconnection of our server connections between our data centers and the internet. And that total loss of connection caused a second issue that made things worse." /9

"To ensure reliable operation, our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection." /10

"In the recent outage the entire backbone was removed from operation,  making these locations declare themselves unhealthy and withdraw those BGP advertisements." /11

"The end result was that our DNS servers became unreachable even though they were still operational. This made it impossible for the rest of the internet to find our servers." /12

"All of this happened very fast." /13

Read the full analysis over here /14
engineering.fb.com/2021/10/05/net…