I'm a former Site Reliability Engineer at Facebook (albeit one from many moons ago).
Here's my modest take on #facebookdown :
Assuming the account of events below is true (it does sound plausible) we could still be looking at a few more hours of outage, here's why ⤵️
1/n
Here's my modest take on #facebookdown :
Assuming the account of events below is true (it does sound plausible) we could still be looking at a few more hours of outage, here's why ⤵️
1/n
https://twitter.com/zkikvidze/status/1445080289643016197
What happened ?
It appears Facebook has inadvertently cut itself off from the rest of the Internet. More accurately, it mistakenly removed every "road sign" worldwide pointing at its network.
The "why" will be a very interesting post mortem to read. How hard is it to fix ?
2/n
It appears Facebook has inadvertently cut itself off from the rest of the Internet. More accurately, it mistakenly removed every "road sign" worldwide pointing at its network.
The "why" will be a very interesting post mortem to read. How hard is it to fix ?
2/n
It depends on 2 factors :
1. Whether there's a *workable* backchannel for remote engineers to access not only the systems themselves, but crucially the comms tools and documentation they need.
2. How well rehearsed of a disaster recovery plan they had for this kind of issue.
3/n
1. Whether there's a *workable* backchannel for remote engineers to access not only the systems themselves, but crucially the comms tools and documentation they need.
2. How well rehearsed of a disaster recovery plan they had for this kind of issue.
3/n
You can reasonably assume they had some sort of emergency out of band remote access set up.
But do they still have access to all the fancy internal comms + incident management tools + documentation right now ?
This is less certain and could slow down remediation hugely.
4/n
But do they still have access to all the fancy internal comms + incident management tools + documentation right now ?
This is less certain and could slow down remediation hugely.
4/n
How often did they run drills for a "network down" situation ? Did they have contact numbers in their phones ? Documentation printouts with them ?
Honestly, an outage like this one is so far fetched that it's unlikely they would have had 100% of bases covered.
5/n
Honestly, an outage like this one is so far fetched that it's unlikely they would have had 100% of bases covered.
5/n
WFH, of course, would have made this worse.
When the tools you rely on for your daily comms with colleagues are unavailable, it adds an extra burden to the already sky high cognitive load of troubleshooting a thorny and high stakes technical issue.
6/n
When the tools you rely on for your daily comms with colleagues are unavailable, it adds an extra burden to the already sky high cognitive load of troubleshooting a thorny and high stakes technical issue.
6/n
Needless to say this is an extremely stressful event for the engineers involved, but Site Reliability / Operations engineers are in it for the adrenaline. They will no doubt remember this day for the rest of their careers. Sparing a thought for my former colleagues !
7/7
7/7
Ok, if the below is confirmed and having also spoken to former colleagues, I'm clearly more pessimistic as to their preparedness for the issues I explained above 😓
https://twitter.com/sheeraf/status/1445099150316503057?s=20
(for my French speaking peeps)
Excellente explication plus technique de @AtaxyaNetwork ici :
Excellente explication plus technique de @AtaxyaNetwork ici :
https://twitter.com/AtaxyaNetwork/status/1445096685286350849?s=20
Ouch. Looks like we have part of our answers regarding remote access still being workable or not :
https://twitter.com/BNONews/status/1445120484270649345?s=20
And this from the excellent @Cloudflare blog.cloudflare.com/october-2021-f…
Now on a lighter note :
Here's a screenshot I made from one of Facebook's internal tools back in the day (~2010, I hear they're more sophisticated now !)
No pressure at all on the SREs 😅
Someone must have pressed OK this morning ? 🙂
Here's a screenshot I made from one of Facebook's internal tools back in the day (~2010, I hear they're more sophisticated now !)
No pressure at all on the SREs 😅
Someone must have pressed OK this morning ? 🙂
(I think I clicked OK that day, YOLO)
Pour les 🇫🇷 : je serai à priori à 20h dans @_Techco_ sur BFM Business pour revenir sur cette panne de Facebook.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
