Thread by @ChrisHanel

Okay, so quick postmortem on the KH2 run. What happened, and how did we recover? WELL.... #AGDQ2022

For a run as complex as this, the runners were using their own tracking software. Those were then connecting to a Heroku server, which passed the data along to a NodeCG instance generating graphics embedded in our main package running at the studio.

Everything was verified as working before the run, and we tested that data was properly being parsed at every step of the journey. The run started, and everything looked great!

However, a few minutes in, the data started to reset in ways that we couldn't explain. The heroku server wasn't restarting, but the check data wasn't being preserved. All hell broke loose at this point.

We had a group discord with @SupportClass devs @Dillon_Pentz and @hoishinxii along with KH2 tracker devs @Red__Buddha and @Tommadness , with @Test_Runner at the GDQ studio.

New checks were being sent properly, but with no memory of what had been sent before. And then after awhile, it would reset again.

We had some manual override controls to update the counts, but not show which items -- and then the server would then override THOSE counts. So we disconnected the heroku server- but got our wires crossed in the process, causing the numbers to disappear entirely.

Once that was addressed, we moved to having @Tommadness and @Red__Buddha monitoring each individual runner's raw feed (where they could see the tracker) and updating the override count...

While @Dillon_Pentz and @hoishinxii would follow messages in discord to literally send console commands directly to NodeCG pretending to be the heroku server, which would trigger the item appearances.

They then did this for about THREE HOURS. At one point, Violin's own tracker crashed and reset, but managed to get back on the same pagem

All in all, this was a fantastic team effort from everyone involved to do some extremely high-pressure crisis management and keep things smooth. Without the trackers, a race like that can be impossible to follow, and I'm extremely relieved we kept things running!

Or, to boil this whole thread down to one tweet:

https://twitter.com/hoishinxii/status/1481814737206022144