Profile picture
Nick Craver @Nick_Craver
, 6 tweets, 3 min read Read on Twitter
@StackStatus We are now stable and are digging into the contributing factors. The outage was ultimately caused by saturating the 20Gb network connection from our SQL Servers but the query generation differences in EF Core were the underlying cause. We're getting to the bottom of it now.
@StackStatus This query was being filtered client-side in EF Core (not SQL-side as it was with Linq2Sql). The .Where(...) was being applied client side after bringing back 52 million posts from SQL Server. This brought us offline. Still digging into why this happens.
@StackStatus While I will dig to investigate what happened here...we're just replacing it with Dapper instead. It's drastically simpler in this case.
@StackStatus For most queries, Dapper is just a better path for us - in this case:
@StackStatus Some back-of-the-napkin math shows we'd have needed about 183Gbps of network bandwidth on the SQL Servers to stay online during that event. Nothing to be solved there, just a "fun" fact. They were dumping straight from memory and saturating the network.
@StackStatus Writing this postmortem is...weird.
"Instead of pulling back dozens of posts from the databases (filtered remotely), the StackExchange app pulled back 142 million posts per user profile load (then filtered locally)."
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Nick Craver
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member and get exclusive features!

Premium member ($30.00/year)

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!