Discover and read the best of Twitter Threads about #rogersoutage

Most recents (5)

I don’t have any inside knowledge of what happened with #RogersOutage, but I do have a ton of experience with control systems, and critical infrastructure … and what happened yesterday is a giant red flag for all of us. /1
In broad strokes, it’s fairly easy to design a system with adequate redundancy to survive a single point of failure - even one at a central point. The control systems world has been doing this with pipelines and electrical grids for decades. /2
Generally, you can lose any one node in a network and recover from it fairly quickly. Start losing many nodes nodes in a network, and things get hairy real fast - this is a “cascading failure”, where whatever triggered the first failure results in multiple failures. /3
Read 19 tweets
#rogersoutage We observed a few rounds of recovery attempts and fraction of traffic came back multiple times. But none of the recovery lasted over one hour. The network went completely down again at around 23:00 UTC. More in the updated blog post: blog.cloudflare.com/cloudflares-vi…
As described in the blog post, we also saw a pattern of #BGP route flapping in the form of large number of announcements followed by withdrawals. This could indicate internal/core network issue getting propagated out via BGP.
The traffic starts to ramp up again! A bunch of curious engineers here (virtually) are watching the traffic monitors and cheer to see higher throughput. It's been 40 minutes now, and it seems to be able to sustain the load this time.
Read 11 tweets
Now this is interesting. From @CloudflareRadar via @jordancrombie, a flurry of BGP announcements right before Rogers went offline.

🧵 for my short story of what might be happening. I had fun writing it.

#ItsAlwaysBGP #rogersoutage #rogersdown
Here's my real guess. Rogers BGP announced itself off the internet, and now isn't successfully peered with anyone. Since they're now not peered with anyone, they can't announce themselves back on.
They have to make phone calls to their peers to get manual routes put in to get them bootstrapped enough to be allowed to announce the correct BGP entries again.
Read 22 tweets
I wonder what they told it? Image
@Grady_Booch why the self declaration for OOP?
If anyone tells you that you can't ignore your problems forever, just remember that they are quitters.
Read 976 tweets
You're job CAN be your life but it doesn't have to be.

I enjoy my job, I'm good at my job, but I do it so I can support my family.
Can Apple and Android get their emoji game in order? I'm getting sick of seeing crossed out boxes
It's odd how often accepting people comes with stipulations.
Read 1019 tweets

Related hashtags

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!