Thread by @EmiT87

(0/11) 🧵 about what happened during weekend and solutions used by @solana team and validators for dealing with these issues 👇

(1/11) Crypto market started crashing (including $SOL) on Thursday afternoon (UTC time), almost at the same time Solana throughput went down significantly marking the start of this slowdown.

(2/11) What followed on? As a result of the rapid decline of SOL value, people which borrowed USD on DeFi platforms like @solendprotocol and used SOL as their collateral got liquidated, because the value of their collateral was lower than 120% of the amount borrowed.

(3/11) What does this mean? Liquidator buys this position, sells whole collateral, returns borrowed money and takes 5% cut which is called liquidation bonus and this is IMO the main reason why a this storm happened.

(4/11) 5% bonus is a really good incentive for bots to automate this process, on Solend these bots made roughly 400K USD in bonuses, according to info from @0xrooter

(5/11) How to be the most successful liquidator on Solana during these peaks? You have to be lightning fast. So you cannot use RPC interface for sending those transactions (smart contract interactions). What they used/exploited?

(6/11) They started sending TXs via Solana’s internal protocol directly to Validator TPU (Transaction Processing Unit) of the current slot leader (which is creating block). TXs are cheap so you rather send multiple TXs (dupes) to be sure to be included in the block.

(7/11) Because leader schedule is well known (and it’s based on amount of stake), these bots targeted the biggest validators. That’s the reason why @certusone, @BlockdaemonHQ @ShinobiSystems @triton_one and other big validators suffered the most.

(8/11) Our @rbf_cap validator is only in the Top 50 biggest validators and still our validator received of 4,6 Gbps incoming TPU traffic, which was blocked by our Anti-DDoS solution.

(9/11) So which changes has Solana implemented in versions 1.8.13 and 1.8.14? Version 1.8.13 stopped forwarding before mentioned TX from TPU to other validators and 1.8.14 improved the performance of signature verification of those dupes.

(10/11) You can see that since 23th of January (1.8.14 release and deployment), network performance is stable (the spike in the graph is our 1.8.14 deployment).

(11/11) It’s great to see validator community solving this problems in #mb-validator Discord. A lot of great new ideas were proposed by the community and we will hopefully see them implemented in upcoming versions. @aeyakovenko