Thread by @heymingwei

#rogersoutage We observed a few rounds of recovery attempts and fraction of traffic came back multiple times. But none of the recovery lasted over one hour. The network went completely down again at around 23:00 UTC. More in the updated blog post: blog.cloudflare.com/cloudflares-vi…

As described in the blog post, we also saw a pattern of #BGP route flapping in the form of large number of announcements followed by withdrawals. This could indicate internal/core network issue getting propagated out via BGP.

The traffic starts to ramp up again! A bunch of curious engineers here (virtually) are watching the traffic monitors and cheer to see higher throughput. It's been 40 minutes now, and it seems to be able to sustain the load this time.

The network recovery process seem plateaued, and it carries less than 20% of what it carried yesterday same time. We still see comparable and number of BGP announcements and withdrawals now, which indicates the core network issue has not been resolved yet.

#rogersoutage Partial recovery. See how much data it carries as compared to 24 hours ago in graphs here.

https://twitter.com/CloudflareRadar/status/1545591714227933186

#rogersoutage Rogers traffic rate remains increasing slowly for the past hour, now reaching 22% comparing to the rate 24 hours ago.

42% now. Rogers is back.

https://twitter.com/TheTannerRyan/status/1545632592485142528

#rogersoutrage Here is a graph showing Rogers' traffic slowly climbing back up for the past 4 hours. I also saw reports of people getting their phone and Internet access back. I hope this means the outage is over.

Seems like #rogersoutage is over.

https://twitter.com/RogersHelps/status/1545724742107267073

Here is graph showing the #BGP message volume for the past 2 days during #rogersoutage observed from @Cloudflare vantage points. The outage started at 08:30 UTC on Friday and ends starting at 00:00 UTC on Saturday (when we start to see traffic ramping up).

@Cloudflare @CloudflareRadar also provides public dashboard for you to keeps tap on how much traffic we are seeing as compared to a week ago. We can see that the traffic has come back to the normal rate now. You can access the graph for Rogers here: radar.cloudflare.com/asn/812?date_f…