Thread by @helium

This week the @helium Network was severely tested. The core developers and Helium community have worked together to address multiple challenges that have resulted from its unprecedented growth.

A more detailed, engineering-specific post-mortem will be published later this week, but we wanted to provide initial details on these incidents so everyone could have some early context on what happened.

First, BIG THANKS to everyone for their contributions and their patience through the past week. A core strength of this Network is that it’s built by the people: Hotspot owners, Validator operators, device owners, developers, manufacturers, and mods make the Network resilient.

Second, during these challenges, $HNT was always safe, and during the disruptions, data transfer was largely unaffected. Robust data transfer was designed into the Helium Network from day one and will be continuously improved and updated by the core developers and the community.

Both last week and this weekend’s outages resulted from large-sized blocks that Hotspots and Validators could not agree upon and ingest into the chain. These blocks were filled with transactions from faulty state channels that process data transfer packets for Discovery Mode.

The Network was able to recover from last week’s chain halt after Validators deployed new fail-safe measures to prevent large blocks from being proposed and prevent new faulty state channels from reoccurring.

These changes were rapidly developed and deployed by core developers and the community. Read the full details on the Engineering Blog: bit.ly/3CF1jdS.

This weekend a second large block made up of remaining Discovery Mode state channels had already formed before the fixes were applied. Helium App developers have since decided to disable Discovery Mode until further notice.

In addition to these changes, new firmware releases were made available to all Hotspot Makers to ensure their Hotspot fleets could sync, participate in Proof of Coverage, and transfer device data. Read about the releases on the Engineering Blog: bit.ly/3CDFtaM.

The community and the HIP-19 process has approved a variety of Hotspot vendors, all of which provide diversity and resilience to the Network. These full blockchain nodes are affected differently by software updates and organic network expansion to 300K nodes and beyond.

The core developers commit to supporting all currently approved Hotspots with core blockchain and miner updates until the Light Hotspot software upgrade is available for manufacturers to implement and migrate their fleets.

These early days of building the Network are challenging. We have learned a lot very quickly over the last two years and have made significant enhancements to ensure all continues to run smoothly and make for a stronger and more resilient Network for the next several decades.