Anatomy of a patch
In the past few hours a critical security vulnerability and patch were disclosed on Solana, this public disclosure occured after a supermajority of stake had already been patched to protect the network. Let's look at how this process unfolded and how 70% of stake were able to be patched before the public disclosure:
First outreach
We were first contacted by multiple different members of the Solana Foundation via various different communication platforms via private message, these are people we know personally and have communicated with before, therefore their identity is known and verified to us.
The first message was received on Wednesday, 7 August 2024 at 14:56 UTC, advising of an upcoming critical patch and sharing a hashed message confirming the date and unique identifier of the incident, the hash shared in this message was published by multiple prominent members of Anza, Jito and Solana Foundation on Twitter/X, Github and even Linkedin in order to confirm the veracity of the message.
The message provided a specific date and time at which to expect receipt of the patch in order to urgently apply this to Mainnet nodes to protect the network, as the patch itself discloses the vulnerability, time therefore being critical once it is first circulated.
Follow-up
During the next 24 hours several other core members reached out to confirm readiness and reiterate the need for urgency and confidentiality.
Patch time
At the pre-determined time, 14:00 UTC yesterday, 8 August 2028, we received a further message, again from two separate Solana Foundation members, containing instructions on downloading, verifying and applying the patch. The patch itself was hosted on the Github repository of a known Anza engineer.
The instructions included verification of the downloaded patch files against supplied shasums and could be manually introspected for their code changes as well.
At no point were operators asked to run a closed-source or private binary.
Through extensive and ongoing outreach by the Solana Foundation, Anza, Jito and others within several hours a superminority and soon thereafter a supermajority (66.66% of stake) was patched. Once 70% was patched the network was ostensibly safe and the existence of the vulnerability and the patch were disclosed in public with a call for all remaining operators to upgrade.
How do you contact validators in a decentralized network?
This question has arisen but it's really not that complicated. Most validators are active on Discord, many are also active in various Telegram groups, we interact on Twitter/X and might even know Anza or Foundation employees personally from Breakpoint etc. It's tedious but not difficult to DM validators in order to pass on such messages, especially with a group of 5-8 core people all participating in this outreach.
The key is to manage to contact enough stake to protect the network while retaining confidentiality. The amazing thing about Solana's validator community is that it's very active and engaged, and even if you don't directly know a validator they're often only one degree of separation away as we've all made friends with others over the years.
The added complexity with this sort of exercise is that operators exist in all time zones, e.g. for us the patch release was at 2am in the morning. Ultimately this is the sort of thing that happens in a complex computing environment, the existence of a vulnerability is not a concern but the response matters, the fact this was caught and safely resolved in a timely manner speaks volumes to the ongoing high quality engineering efforts that are often not visible to the public, by Anza and Foundation engineers but also engineers at Jump/Firedancer, Jito and all the other core contributing teams.
In the past few hours a critical security vulnerability and patch were disclosed on Solana, this public disclosure occured after a supermajority of stake had already been patched to protect the network. Let's look at how this process unfolded and how 70% of stake were able to be patched before the public disclosure:
First outreach
We were first contacted by multiple different members of the Solana Foundation via various different communication platforms via private message, these are people we know personally and have communicated with before, therefore their identity is known and verified to us.
The first message was received on Wednesday, 7 August 2024 at 14:56 UTC, advising of an upcoming critical patch and sharing a hashed message confirming the date and unique identifier of the incident, the hash shared in this message was published by multiple prominent members of Anza, Jito and Solana Foundation on Twitter/X, Github and even Linkedin in order to confirm the veracity of the message.
The message provided a specific date and time at which to expect receipt of the patch in order to urgently apply this to Mainnet nodes to protect the network, as the patch itself discloses the vulnerability, time therefore being critical once it is first circulated.
Follow-up
During the next 24 hours several other core members reached out to confirm readiness and reiterate the need for urgency and confidentiality.
Patch time
At the pre-determined time, 14:00 UTC yesterday, 8 August 2028, we received a further message, again from two separate Solana Foundation members, containing instructions on downloading, verifying and applying the patch. The patch itself was hosted on the Github repository of a known Anza engineer.
The instructions included verification of the downloaded patch files against supplied shasums and could be manually introspected for their code changes as well.
At no point were operators asked to run a closed-source or private binary.
Through extensive and ongoing outreach by the Solana Foundation, Anza, Jito and others within several hours a superminority and soon thereafter a supermajority (66.66% of stake) was patched. Once 70% was patched the network was ostensibly safe and the existence of the vulnerability and the patch were disclosed in public with a call for all remaining operators to upgrade.
How do you contact validators in a decentralized network?
This question has arisen but it's really not that complicated. Most validators are active on Discord, many are also active in various Telegram groups, we interact on Twitter/X and might even know Anza or Foundation employees personally from Breakpoint etc. It's tedious but not difficult to DM validators in order to pass on such messages, especially with a group of 5-8 core people all participating in this outreach.
The key is to manage to contact enough stake to protect the network while retaining confidentiality. The amazing thing about Solana's validator community is that it's very active and engaged, and even if you don't directly know a validator they're often only one degree of separation away as we've all made friends with others over the years.
The added complexity with this sort of exercise is that operators exist in all time zones, e.g. for us the patch release was at 2am in the morning. Ultimately this is the sort of thing that happens in a complex computing environment, the existence of a vulnerability is not a concern but the response matters, the fact this was caught and safely resolved in a timely manner speaks volumes to the ongoing high quality engineering efforts that are often not visible to the public, by Anza and Foundation engineers but also engineers at Jump/Firedancer, Jito and all the other core contributing teams.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
