I'm not surprised to find out that Linux kernel socket load balancing is in a terrible state with no good options. Some background:


Traditional epoll wakes every thread and they race to accept the connection. EPOLLEXCLUSIVE fixes it but uses LIFO order.
LIFO order is terrible for a web server. HTTP connections are generally long-lived and reused for mixed / varying workloads. That's even more true with HTTP/2 where clients are only supposed to make a single connection to each server and multiplex everything over it concurrently.
In a standard nginx setup on Linux, it uses EPOLLEXCLUSIVE. This gives nearly all the connections to the same worker until it starts getting overloaded. Even then, the most overloaded workers still keep getting the most connections among handling other events. It's pretty awful.
The alternative that's offered is using reuseport to have the kernel evenly distribute new connections across workers.

It doesn't account for the varying usage of connections. It happily hands out new connections to a worker that's not even idle. It's not as nice as it sounds.
The kernel developers were gradually improving REUSEPORT in different ways. However, they stopped and allowed some of it to regress. You're supposed to use BPF to give the kernel a load balancing approach specific to the application. Of course, applications don't actually do it.
A decent approach for many applications would be if epoll used FIFO instead of LIFO order. It would work like reuseport but only distributing connections to idle workers. It was proposed in 2015 with EPOLLEXCLUSIVE but didn't land. It's apparently what Cloudflare uses themselves.
marc.info/?l=linux-fsdev… was an attempt at reviving it. Cloudflare tends to not be persistent enough to get through the hassle of getting changes landed in the Linux kernel or nginx.

They deal with the initial technical aspect but not the politics / evangelism to get it landed.
They also tend not to invest resources into making things into a more generic solution suitable beyond their use case. Makes sense for them. They clearly don't see that much value in their changes being upstream. The upstream projects tend not to care much about latency, etc.

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Daniel Micay

Daniel Micay Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @DanielMicay

2 May
Likely going to have very positive news to announce about GrapheneOS in the near future.

It's unfortunate that others decided to pick up the torch of fighting an underhanded war against us after Copperhead started to stop. The past month SHOULD have been a huge relief, but no...
Nevertheless, there's probably going to be some great news. I hope others realize that waging a misguided war against us is not going to work out to their benefit and they can either stop or destroy themselves in the long run. I'll happily put very substantial resources into it.
One consequence of all the nonsense with Copperhead was lack of time to hire people. Now we have a lot of money we need to spend along with a lot more time and resources being freed up. I'll happily spend a substantial amount paying people to counter any further shenanigans.
Read 9 tweets
2 May
Despite previously claiming to be my friend and a supporter of my work, @nickcalyx has consistently engaged in trying to portray me as being deranged/crazy. He has participated in persistent harassment and bullying targeting me. It's far worse than anything Copperhead ever did.
I was naive and repeatedly tried appealing to him for help against attacks from Copperhead and then the even worse behavior from his own community.

Copperhead has stopped their attacks, at least for the time being, but Calyx has continued carrying it on and it's worse than ever.
Calyx took full advantage of what Copperhead was doing and tried to use it as a way to benefit themselves. They actively engaged in making it worse while playing the both sides angle. They've now done things far worse than anything Copperhead did in years of trying to hurt us.
Read 6 tweets
2 May
I'm not sure why I didn't make this change sooner:


There's not much downside to only having 1 slot per slab once slab allocations are 4096 byte aligned, which is what we refer to as extended size classes. This goes nicely with the guard slab feature.
By default, there's an unused guard slab between every possible usable slab that's left as unused PROT_NONE memory.

If the slabs have 1 slot, the allocations have guaranteed guard pages without paying the cost of actually making system calls to set them up during regular usage.
Having slabs with 1 slot originally wasn't supported but github.com/GrapheneOS/har… added support for it.

This does need a tiny little bit of unnecessary work but avoids other work (bitmaps). It's insignificant for allocations this large, especially with zero-on-free, etc. enabled.
Read 6 tweets
21 Feb
We've archived these tweets where Copperhead's CEO admits to them tracking devices via unique identifiers and using them as part of the update system.

He admits that their phone sellers with Copperhead emails, etc. have databases mapping unique identifiers to the customers too.
His excuse is that tracking devices via unique identifiers available to update server doesn't count as tracking users. They've designed it in a way that they can ship an update targeting a device. The excuse is they don't know which user has which device, but their seller does.
In that same thread, he also peddles the usual lie that Copperhead is source available. Meanwhile, the sources are not published and are not available for review. Multiple researchers including a Whonix developer have attempted to get access to the sources and could not get it.
Read 8 tweets
19 Feb
@HamledOnLine @MikeCustomPC @cankerwort_ @sethisimmons @CopperheadOS @mamushi_io @GrapheneOS We still have the original repositories created for CopperheadOS Beta and later. That was before the company was founded too. I created the project on my own time substantially before the project was founded as an extension of my existing hardening work.

@HamledOnLine @MikeCustomPC @cankerwort_ @sethisimmons @CopperheadOS @mamushi_io @GrapheneOS I started github.com/GrapheneOS/har… after the split while figuring out how to put things back together for the OS. I waited until the release of the next major version of Android to make the next release under the temporary Android Hardening name, and then renamed to GrapheneOS.
@HamledOnLine @MikeCustomPC @cankerwort_ @sethisimmons @CopperheadOS @mamushi_io @GrapheneOS See github.com/yegortimoshenk… for an archive of some of what happened as part of the takeover attempt. There are other archives of earlier code and information. James tried to get this repository wiped out with bogus DMCA claims and threats against @yegortimoshenko.
Read 7 tweets
19 Feb
@cankerwort_ @sethisimmons @CopperheadOS @mamushi_io @GrapheneOS They're the ones choosing to a misinformation war against GrapheneOS along with threatening/intimidating anyone who contributes to the project, even people that are underage. You say it should be settled in court but they're making daily attacks on us causing lots of harm.
@cankerwort_ @sethisimmons @CopperheadOS @mamushi_io @GrapheneOS GrapheneOS not a for-profit project. We're not selling any products. We're focused on building privacy and security technology. They're focusing all their resources on causing harm to us in any way that they can, and on marketing a product simply copy pasting our codebase.
@cankerwort_ @sethisimmons @CopperheadOS @mamushi_io @GrapheneOS This is primarily not a legal dispute. It's largely a personal vendetta against me by James Donaldson and now also Max. Their poorly formed lawsuit against was primarily a way to exhaust my time, energy and resources along with intimidating people to stop them from contributing.
Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!