, 15 tweets, 3 min read Read on Twitter
Rare evening technical thread: about distributed systems, exponential backoff, and the tragedy of the commons problem.
Let's say you're running some kind of service that clients call, and it has an "availability event" ... which is, you know, PR speak for "outage". Anyway, so you have this event and you fix the problem, but that doesn't always mean that you recover right away.
Often your clients will have started to retry their requests. Clients build up over over time, more and more of them. You can easily get into a situation where you now have way more requests than you ever ordinarily would. And this just as you are trying to recover. Uggh.
If you have some kind of throttling, you can apply that, but the best strategy is actually to have clients backoff. @clare_liguori has gone over this very well so read what she has to say ...
The main takeaways are to do exponential backoff and add some random jitter. Do that and the recovery will go more quickly. O.k. so what am I tweeting about?
There's a challenge with this kind of backoff; it works well only if everyone, or at least "nearly everyone", does it. A single client could decide to be more greedy and retry more aggressively - and it's sort of in their interest.
If everyone else is well behaved, that greedy client will probably get service more quickly, they'll jump the queue basically. But the overall recovery will be slowed down a bit by the aggressive tactics. Then, if this behavior spreads, it's disaster.
Everyone retrying aggressively makes the whole situation worse, for everyone. It's a "tragedy of the commons" - everyone acting in their own narrow self-interest actually makes everyone worse off in the end.
I often think of airport baggage belts, where people compete and huddles to get closer to the belt; we'd all be better off if we *all* just stood back a few meters. Clearer views, easier access, easier exit.
Many distributed systems really rely on this kind of community oriented cooperation. TCP, BGP, the Web, e-commerce, Cloud Services. All have examples where a kind of codified socialism, fairness and restricting individual competition, keep the systems safe and cost-effective.
It needn't be like that; industries don't have to co-operate! Stock markets are the opposite, where there's a costly (maybe wasteful) race to be the fastest and try very aggressively, producing HFT.
Blockchains are another example, where there is deliberate aggressive competition, and it's even exploited to drive a perverse consensus based on mutual mistrust.
Anyway, back to the tragedy of commons! Afaik, no-one has really found a robust solution to this beyond blocking mis-behaving clients before they spread, which also comes at expense.
IME, how it "really" works is that the folks who builds clients and services have a sort of code of honor system. Everyone recognizes that it would be a wasteful race to the bottom, and mostly self-policing works. So browsers, SDKs, etc ... all do sane, safe, decent things.
I don't really have a big technical lesson there, I just find it fascinating that huge sections of the economy can get by like that. It's very inspiring and reassuring! /EOF
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Colm MacCárthaigh
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!