Thread by @rahaeli

The Twitter mentions tab has never been truly reliable, but like the embedded tweet Archane is quoting says, it's gotten much worse lately. From checking my tweets directly I'd estimate I'm seeing less than 30% of my replies.

https://twitter.com/archane/status/1614393280812429312

People don't realize that's happening, because it's more reliable if you only get a few notifications. (I'll explain why in a second: short answer, services degrade at volume.) I only have about 10k followers; people with more are having a much worse time of it.

"Don't take it personally if people don't reply" was always a good rule for Twitter, but it's crucial now. The person you're talking to almost certainly didn't see the reply you think they're ignoring. This is why I like every reply to me I see: so I know I've seen it.

To explain why this happens, let's talk architecture. It's not efficient for a service to look up every post you've made every time you load your notifications and count whether someone interacted with it every time you want your notifications.

That's too much database load, even if you cache it. Instead, you run a separate process (usually called a worker or a service) that you basically send a memo to whenever someone interacts with a post (likes, retweets, comments, etc) in a way that should notify you.

You interacting with the post sends the service a memo that an interaction happened. That service takes that memo and puts a note in a separate part of the database that says "Susie interacted with Mike's post: like" or "Susie replied to Mike".

Then, when Mike loads his mentions, the site asks that database table "okay, what are the most recent memos about interactions?" It may do some batching ("X people liked your tweet"), or it may show them individually; it depends on the site.

The notification logging service can get behind on sending its memos, though. An example is on DW: when we make a news post, so many people get email notifications of it that it can take hours for the email sender to chew through them all.

(And we've had a perpetual issue with the email sender worker getting so overwhelmed that it forgets where it was in the to-do list, gets distracted, and starts over at the beginning of the batch, even if some already were sent: the source of the duplicate email problem.)

Depending on how the notification service is made, getting overwhelmed may mean a delay but 99% of the notifications get there eventually, or it may look like what's happening with Twitter: if the worker gets backed up, it just drops older memos to concentrate on the newer ones.

That's what looks to be happening with Twitter. The notifications service is getting overwhelmed more easily, because Elon has cut the resources available to that service, so it just throws out its backlog so it can keep accepting memos for newer interactions.

This is why there's no real rhyme or reason to which notifications you miss getting: it depends entirely on how overloaded the service was *at the time the notification event was fired*. It can come down to "it was overloaded for 2 seconds and dumped everything in that window".

Sites run multiple instances of these servers in order to parallelize the work (the more desks you have to put memos on, the less likely the memos will slide off the desk into the trash).

The more notifications you're getting, the more often a notification job for your account will go "I'm too overwhelmed, I'm dropping this one", while if you only get a few, the chance of you hitting a drop-out is lower.

Cutting resources to the notification service, which is what Elon has done, means the service is more likely to drop stuff on the floor, just like firing half a call center means there's more chance an incoming customer phone call won't connect.

Unlike in a call center, though, the difference is a lot of sites don't have a long "hold queue" for notifications. If there's no room for the memo on someone's desk, it just gets tossed in the trash: equivalent to you catching up with task backlog by throwing out the oldest.

And there isn't really a way to go back and catch up with work later. If the memo didn't get written at the time the event happened, or it got written but thrown out and the cleaners emptied the trash, you can't go back and recreate it.

So, tl;dr: notifications are very broken right now, it's getting worse instead of better, and the larger an account is, the more likely they'll lose notifications because the notification service was overwhelmed when you replied and the memo about it is sitting in a landfill.