1/ Crypto markets experienced extreme volatility over the last 24 hours, with significant drawdowns in crypto asset prices across the board.
The excessive volatility induced stress on Anchor, causing several collateral effects.
The excessive volatility induced stress on Anchor, causing several collateral effects.
2/ Beginning around 9:36 PM KST, the LUNA price had retraced to roughly 9.8 UST.
The sharp decline in the price of LUNA induced many Anchor users with open borrowing positions to access the Anchor web app to pay down their loans and avoid liquidation.
The sharp decline in the price of LUNA induced many Anchor users with open borrowing positions to access the Anchor web app to pay down their loans and avoid liquidation.
3/ Anchor’s Cloudflare dashboard reported 2.58 million requests in 30 mins to Anchor Mantle, creating resource starvation on Mantle.
Anchor Mantle nodes were overloaded, unable to accept new requests, and became unresponsive, causing an unresponsive Anchor web app for 30 mins.
Anchor Mantle nodes were overloaded, unable to accept new requests, and became unresponsive, causing an unresponsive Anchor web app for 30 mins.
4/ Within 30 mins, a fix was deployed to the Anchor_Mantle cluster to replenish its resources.
While still slow at the time, the web app was presenting the correct numbers as the smart contracts on-chain were working correctly.
While still slow at the time, the web app was presenting the correct numbers as the smart contracts on-chain were working correctly.
5/ LUNA’s sharp price decline triggered 4,000+ liquidation transactions in that time span.
Out of the 9,542 outstanding loan positions on Anchor, at least 800 loans were at risk of liquidation.
Out of the 9,542 outstanding loan positions on Anchor, at least 800 loans were at risk of liquidation.
6/ The result was massive network congestion, where transactions were no longer being accepted by the RPC nodes.
Congestion-induced side effects included:
Congestion-induced side effects included:
7/
1. TX broadcasts resulting in timeout impacted oracle feeds leading to missing votes, which turned off the on-chain swap feature due to invalid prices.
2. An Oracle Feeder transaction that was accepted by the network triggered another wave of liquidation transactions.
1. TX broadcasts resulting in timeout impacted oracle feeds leading to missing votes, which turned off the on-chain swap feature due to invalid prices.
2. An Oracle Feeder transaction that was accepted by the network triggered another wave of liquidation transactions.
8/ The Anchor team has come up with a series of preventative measures and fixes to counter the above issues from occurring again during periods of extreme market volatility.
9/ First, Anchor is currently migrating to a more scalable back-end infrastructure to avoid the congestion and resource depletion experienced above.
Anchor back-end has already scaled up to handle even more requests than what induced the chain of events over the last 24 hours.
Anchor back-end has already scaled up to handle even more requests than what induced the chain of events over the last 24 hours.
10/ Second, the following solutions to alleviate severe congestion induced by liquidation and transaction spamming are being weighed:
11/
1. Swapping the “take-all” mempool logic to a price-sorted optimistic liquidation trigger that would balance the number of txs and gain from liquidation.
2. Implementing an on-chain mempool logic fix where txs are ordered by gas fee, then max block gas limit.
1. Swapping the “take-all” mempool logic to a price-sorted optimistic liquidation trigger that would balance the number of txs and gain from liquidation.
2. Implementing an on-chain mempool logic fix where txs are ordered by gas fee, then max block gas limit.
12/ Of note -- in Columbus-5, improved gas metering by CosmWasm will execute and apply priorities by message (Oracle votes first) by block to prevent the same problem from happening again.
Gotta love mempool prioritization.
Gotta love mempool prioritization.
13/ Finally, in regards to questions about the Anchor deposit yield at 18% instead of 20%, the protocol worked precisely as intended.
Anchor’s APY is intended to stick between 18% (Threshold limit) and 20% (target limit).
Anchor’s APY is intended to stick between 18% (Threshold limit) and 20% (target limit).
14/ 20% nearly always holds as the deposit rate since the utilization ratio remains high as borrower levels exceed deposit inflows.
During extreme LUNA price volatility, borrower rates decline, triggering Anchor to tap into the yield reserve to maintain a minimum of 18% APY.
During extreme LUNA price volatility, borrower rates decline, triggering Anchor to tap into the yield reserve to maintain a minimum of 18% APY.
15/ During normal market conditions with a healthy utilization ratio, excess yield is drawn into the yield reserve.
Currently, the yield reserve is being tapped to maintain the 18% minimum deposit rate.
Currently, the yield reserve is being tapped to maintain the 18% minimum deposit rate.
16/ Of note, based on retracting borrower rates during volatility, deposit rate would’ve been 10 - 11% without the yield reserve buttressing the threshold rate.
With utilization ratio at ~50%, Anchor will dip into the yield reserve until parity is restored and ratio increases.
With utilization ratio at ~50%, Anchor will dip into the yield reserve until parity is restored and ratio increases.
17/ The 18% threshold rate and 20% target rate are also governance-controlled parameters, which can be changed by community vote.
More details on the market parameters below:
docs.anchorprotocol.com/protocol/ancho…
More details on the market parameters below:
docs.anchorprotocol.com/protocol/ancho…
18/ In conclusion, the extreme downward volatility in the price of LUNA caused the Anchor web app to be unresponsive for ~30 minutes before being patched.
All functionality on the smart contract side was operational and the yield reserve backstopped the threshold deposit rate.
All functionality on the smart contract side was operational and the yield reserve backstopped the threshold deposit rate.
19/ Silver lining — this stress test is exceptionally challenging to replicate through testing & was managed w/ no systemic risk to the platform.
Without such challenges, it’s difficult to reason about black swan events & their potential impact on various levers of the protocol.
Without such challenges, it’s difficult to reason about black swan events & their potential impact on various levers of the protocol.
20/ We have taken explicit steps to fix the issues causing the events iterated above, are implementing preventative measures to eliminate the possibility of it happening again, and will improve upon what we’ve learned in the process.
21/ Moving forward, Anchor will be significantly more robust to the events that occurred earlier today. 💪
Thank you for your patience and understanding.
Thank you for your patience and understanding.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
