Thread by @egerlach

Now this is interesting. From @CloudflareRadar via @jordancrombie, a flurry of BGP announcements right before Rogers went offline.

🧵 for my short story of what might be happening. I had fun writing it.

#ItsAlwaysBGP #rogersoutage #rogersdown

Here's my real guess. Rogers BGP announced itself off the internet, and now isn't successfully peered with anyone. Since they're now not peered with anyone, they can't announce themselves back on.

They have to make phone calls to their peers to get manual routes put in to get them bootstrapped enough to be allowed to announce the correct BGP entries again.

So why are they still down? I'll bet one of two things has happened here.

1) They've got themselves in a situation where the bad routes that are out there are trumping the good routes they're trying to put out.

Or, more likely IMO....

(Disclaimer: the rest of this thread contains a fictional depiction of the Rogers Corporation as having a toxic management culture. I have no knowledge of their culture. For all I know it's the complete opposite. I just had some fun with these characters.)

2) There's only one guru at Rogers who knows BGP in and out. He's up at his Muskoka cottage this week, taking time off after Canada Day.

He's been watching countries take themselves off-line for the last 5 years, and been stressing how important it is to have coverage for him.

2 con't) His manager knows that's the case, but people who are good at this don't come cheap. They're some of the most expensive talent out there. The VP didn't want to invest the $300k+ salary it would take to hire someone like this away from another company.

2 con't con't) "It's so unlikely that this will happen, right?" said the VP, "Just don't make any mistakes!"

The guru and the manager advocated for investing time in emergency procedures, negotiating them with their peers in advance, and ensuring they have a step-by-step plan.

2 con't con't con't)

"How would that affect the timeline on project X?" asked the VP.

It would delay it, of course. Project X directly impacts revenue, you see. So that's where the investment was made.

The guru and the manager hoped they would get through another year...

2 con't con't con't con't)

The guru is at his cottage. Off the grid. The manager has ALL the VPs breathing down their neck now. She calls around to every one of the guru's work friends to find out if one of them has ever been to his cottage.

2 con't^5)

Finally, she finds someone at work who is a mutual friend with someone who has been to his cottage. She calls the mutual friend. The mutual friend stays on the phone with the taxi driver they have hired from the closest cab company, 45 minutes away.

2 con't^6)

After the friend gives a few bad directions (he was in the passenger seat after all) the taxi arrives at the cottage. The guru is on the deck, in his boxers, with a beer in hand. His happy place. The driver asks him to turn on his phone, it's an emergency.

2 con't^7)

The guru instantly gets a migraine. He has an inkling of what is happening if this is how work needs to get hold of him. He turns on his phone. It instantly rings.

"Yeah."
"Isn't this exactly what I said would happen?"
"Fine."

2 con't^8)

He gets dressed, and gets in his car for the 3 hour drive back to the office. No way he can remote in, what with Rogers having told the internet it doesn't exist.

On the way, he calls up a few of his contacts at their peers. They laugh at him lovingly. They know.

2 con't 9)

He gets back to the office and assesses the damage. He finally starts the long, tedious job of bootstrapping an Autonomous System.

2 con't 10)

The next day, after he resolves everything, he drives back to his cottage, and keeps his phone on. He knows what's coming.

In the early afternoon, he's summoned to a meeting with the VP and his boss, the SVP. He dials in from his cottage, much to their chagrin.

2 con't 11)

He patiently and kindly explains that this is exactly what he said would happen, and that the plan he and the manager put together is still available. They can give the green light now and then the company will be able to avoid this in the future.

2 con't^12)

He's thanked for his help and his thoughts. He leaves the call to his manager and the VPs.

2 con't^13)

2 hours later, his manager calls him back. She says the VPs aren't investing in his plan. Instead, they're modifying his role so that he's on call 24/7/365, but only for this kind of incident.

He asks what his pay increase would be for this.

Nothing. He's exempt.

2 con't^14)

"If that decision is final, I think I'll start applying for work elsewhere. Someone out there will be looking for an expert who can prevent what happened at Rogers", the guru says.

"But you can't leave, then we'll have no coverage at all...", the manager replies.

2 con't^15)

"That sounds like a problem for you and the VP. I'll get paid more if I leave, and *I'll* be the backup.", the guru replies.

"I'll let him know", the manager says before hanging up.

The manager opens LinkedIn and starts to update her profile...

FIN.

Thank you for reading my short story, "Possibility 2", including a fictional depiction of Rogers as having a toxic management culture.

I have no idea if that's true. But I've seen VPs delay reliability for revenue.

It works for a while...

But you have no safety net.

End 🧵