🧵 An attempt to summarize what happened today.

forking event - this was the initial tweet I and others put out, because this is what we observed, excessive forking. This is however a symptom, not a cause. We don't know the exact root cause yet, but what we know is that over the past week all validators were encouraged to upgrade from 1.13.x to 1.14.16 which is a big jump with a lot of code changes.

Interested in understanding the hardware requirements of Solana validators better?

Will break down the various components and why they matter in this thread

If you want to run a validator or RPC node reach out to @latitudesh (or direct to @guisalberto) for some good deals

🧵 There's always much talk about the hardware requirements for Solana validators.

Why are they so high? Simply put: because Solana wants to achieve the highest throughput that is technologically possibly given the best computing and networking resources.

Possibly final thread/tweet on this for today:

Thoughts on the validator that caused this outage.

🧵 Hopefully it was unintentional, and ultimately it was a code bug that didn't correctly handle the duplicate block scenario (most of the times it is handled correctly but this was a very rare edge case).

A validator ran two instances and propagated two conflicting blocks.

The Solana Mainnet cluster has successfully restarted a few minutes ago, new blocks are being finalized.

So what exactly happened? We don't know everything yet, but here's some of what we do know

🧵 At approximately 2241 UTC on 30 September many validators stopped voting.
Many operators were immediately alerted and converged on Discord to assess the situation.

It was not immediately clear what was happening, it appeared the network halted but there were also signs it hadn't

🔥 STAKEWIZ NEW FEATURE ALERT 🎉

See your STAKING REWARDS & APY graph for the entire history of your stake accounts!

Connect your wallet, enter your wallet address or just bookmark the URL!

Just add your wallet address to the end of stakewiz.com/stakes/

More features 🧵👇 That's right, you can navigate directly to the stakes for any wallet with a direct link!

Here's a random whale for instance:

stakewiz.com/stakes/3JkCkwj…

Go look at their APY, we calculate a TrueAPY by excluding undelegated amounts such as rent & looking at accurate epoch duration

🧵

Facts with references, make of them what you want.

As few as 3 entities control over 50% of ETH validators

(Source: beaconcha.in/pools) The most severe form of censorship is soft-forking, where blocks that contain undesired txs are excluded and forked.

This requires 51% of validators to achieve (quoting @VitalikButerin from the ETH call 5 days ago: )

A look at centralization concerns on Ethereum Beacon Chain.

This thread is an attempt at reviewing facts and gaining understanding, not an attack on ETH, we believe in a multi-chain future.

🧵👇 31.9% of validators are held/controlled by Lido.

Another 32.38% is controlled by just 4 CEXs (Binance, Coinbase, Kraken and Bitcoin Suisse).

An "unknown" pool controls 12.3%.

Those 6 entities/pools control the supermajority of validators (66%+) !!!

beaconcha.in/pools

How many validators does Solana have?

Stakewiz displays over 2050!! Other sources say 1700-1800. Why the difference?

We include all validators who have voted this epoch, even if unstaked OR have at least 1 SOL in active stake, even if delinquent/haven't voted. Why?

If a validator is voting, even with zero stake, their votes are being seen and processed, and we assume they have an intention of receiving stake. If they stop voting and don't have stake they'll drop off our list next epoch.

They're also paying voting fees.

Solana halt today, what happened?

A bug relating to durable nonces was triggered on a majority of validators, leading to a block hash mismatch & validators stalling.

This was actually a known bug that was being fixed and hadn't been triggered in this form previously. The issue is completely unrelated to clock drift and the clock drift has nothing to do with the network halting.

As soon as validators observed that the network ceased producing blocks they started confirming with each other on Discord that everyone was seeing the same thing...

What a weekend.

Yesterday the Solana blockchain halted, which means it stopped producing blocks. This resulted in validators coordinating a cluster restart which requires 80% of stake (minimum 605 validators).

A 🧵 on what occured. 👇 For months we've seen congestion issues on Solana.
Source: NFT and arb/liquidations bots.

When a new project mints bots try to snipe the mints and immediately relist on secondary market places for $$.

The past two weeks has seen an escalation in bot activity...

A massive NFT Drop just hit the Solana network, a drastic increase in CandyMachine txs and associated network spike at 2200 UTC.

A network outage or interruption at Hetzner Falkenstein has also hampered things with ~% of stake being there.

How does it affect validators? 👇 Massive inflow of traffic, not today but on Tuesday for a similar event we saw >20gbps spikes.

This results in A LOT to do for the CPU (even with 24 cores). Deduplicating all those tx, verifying their signatures, then trying to pack a block.

This consumes RAM! A lot of it...

Solana: So 1.8.14 shipped over the weekend, TPS has stabilized. Is everything hunky dory now? Well, not quite yet... While the changes in 1.8.14 have caused massive improvement in reduction of duplicate tx there is more to do, updates in this 🧵 v1.8.14 improved de-duplication of transactions. Solana uses UDP to propagate tx between nodes, unlike TCP with UDP one can't be sure whether the packet was received. As a result transactions are routinely rebroadcast. This has benefits and drawbacks...

Quick 🧵 on our understanding of the current issues & fixes: Due to market volatility many leveraged positions on DeFi become eligible for liquidation. Liquidation can be triggered by anyone on an eligible position, if one liquidates an eligible position one receives a bounty... This creates a market where liquidators race to close eligible positions and get the bounty. Some have created bots to automate this and improve their chances of success. To ensure they get in first and "win" the race they submit the same transaction dozens or hundreds of times..