(a) we had a particularly type of logging turned on
(b) do to a bug in a maintenance process, Cloudflare accidentally routed logs to one of their shut-down servers
(c) this triggered their DDOS systems
4) How we reacted:
(a) we quickly found out that Cloudflare was blocking users
(b) after about 10 minutes, we began switching systems over to a different DNS provider
(c) at the same time, we began talking with Cloudflare's team to diagnose the issue
5) After about an hour, we fully switched FTX's DNS provider, and service was no longer interrupted.
After a few hours, Cloudflare identified and patched the issue; a few hours later, we switched back to Cloudflare.
6) I'd like to thank the @Cloudflare team for being super responsive and helpful, and patching this same-day.
Going forward, we have more practice switching DNS providers and should be able to do this more quickly if there are future issues.
7) Finally I'd like to apologize to Cloudflare and to our developers for the extent to which I'm sure I've botched this explanation.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
Ok; let's say that your goal is to build the most ambitious plausible DeFi ecosystem: one that could end up with a substantial fraction of the world's economy on it.
a) challenge trials fix this
b) safety can be determined with ~1000 people; either you think COVID death >> 0.10% or the whole thing is stupid
c) efficacy was strongly expected
d) fine pay for 40k, it's a tiny fraction of the delay cost
3) On politics:
Both parties get failing grades here. They both flip flopped (remember the immigration wars a year and a half ago?). We transitioned from incompetent to poorly-thought-out.