1/ We are undergoing an outage due to multiple block propagation issues. Heavy spam traffic in the past 30 hours has halted the network.
Our core team is working with validators & internal leader nodes to resolve the outage.
Minimum gas upped to 30 gwei to prevent spam attacks
Our core team is working with validators & internal leader nodes to resolve the outage.
Minimum gas upped to 30 gwei to prevent spam attacks
2/ Progress Update - 1 AM PST
Validators have been advised to downgrade their node binaries to a previous version as the bandwidth of validators with V4.3.2 were overburdened.
Our team is working on a special version of validator node binary to reduce bandwidth usage
Validators have been advised to downgrade their node binaries to a previous version as the bandwidth of validators with V4.3.2 were overburdened.
Our team is working on a special version of validator node binary to reduce bandwidth usage
3/ Progress Update - 1:30 AM PST
Our team is internally testing new versions of Shard 0 that could potentially restore the blockchain.
Our team is internally testing new versions of Shard 0 that could potentially restore the blockchain.
4/ Progress Update - 3:00 AM PST
Validators on Shard 0 and 1 were provided with a new version of the node binary by the Harmony Core team.
Network remains halted but bandwidth usage has reduced.
Validators on Shard 0 and 1 were provided with a new version of the node binary by the Harmony Core team.
Network remains halted but bandwidth usage has reduced.
5/ Progress Update - 5:00 AM PST
Our team was on an active call discussing the issue and its solutions. The team has also designed a specific binary to combat spam.
Important note: The spam that has halted the network are NOT of arbitrage trading bots.
Our team was on an active call discussing the issue and its solutions. The team has also designed a specific binary to combat spam.
Important note: The spam that has halted the network are NOT of arbitrage trading bots.
6/ Progress Update - 5:40 AM PST
Our team is changing internal policies around running new builds on internal nodes.
More information on this will be out in the official postmortem.
Our team is changing internal policies around running new builds on internal nodes.
More information on this will be out in the official postmortem.
7/ Progress Update - 6:30 AM PST
Our team has successfully installed a new build on internal servers in specific regions.
Good news: Outbound spam traffic reduced by about 50%.
The new build shall be installed in additional servers in more regions.
Our team has successfully installed a new build on internal servers in specific regions.
Good news: Outbound spam traffic reduced by about 50%.
The new build shall be installed in additional servers in more regions.
8/ Progress Update - 6:45 AM PST
Our team critically monitored the build to spot any issues before rolling it out to validators.
Good news: Current results showed high promise.
Our team critically monitored the build to spot any issues before rolling it out to validators.
Good news: Current results showed high promise.
9/ Progress Update - 7:30 AM PST
To reduce logging, a change was made to the binary. This has been deployed to Shard 0 and 1 RPCs.
Shard 2 and 3 are being monitored as they haven’t faced an outage yet.
To reduce logging, a change was made to the binary. This has been deployed to Shard 0 and 1 RPCs.
Shard 2 and 3 are being monitored as they haven’t faced an outage yet.
10/ Progress Update - 7:45 AM PST
The new build was installed on internal validator nodes.
Constant monitoring shows reductions in traffic.
The new build was installed on internal validator nodes.
Constant monitoring shows reductions in traffic.
11/ Progress Update - 9:00 AM PST
New build has been released to validators on Shard 0 and 1.
Blockchain can run but awaits the achieving of 66% consensus.
External validators on blocks 0 and 1 were executing the upgrade as the consensus stood at 53%.
New build has been released to validators on Shard 0 and 1.
Blockchain can run but awaits the achieving of 66% consensus.
External validators on blocks 0 and 1 were executing the upgrade as the consensus stood at 53%.
12/ Progress Update - 9:30 AM PST
Shard 0 achieves consensus. But, consensus is unstable and more validators are needed on shard 0 on the update.
Shard 1 hasn’t moved yet.
External validators on shard 0 and 1 MUST update their nodes to the newest build.
Shard 0 achieves consensus. But, consensus is unstable and more validators are needed on shard 0 on the update.
Shard 1 hasn’t moved yet.
External validators on shard 0 and 1 MUST update their nodes to the newest build.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
