Thread by @defunkt

It was a gorgeous Sunday morning. Birds were chirping and squirrels were squirreling. Coffee in hand, I began upgrading GitHub's testing infrastructure. When I was done, I ran a quick test and... deleted the entire production database.GitHub.com

Here's the blog post about the outage: github.blog/2010-11-15-tod…

There were a lot of things that went wrong: Our tests should not have had access to production. Our production DB shouldn't have been wipeable. We should have been able to restore the DB faster. We should also have known our `Events` table would be a doozy...

But the main thing that went wrong: our GitHub application assumed it was running in 'production' mode unless told otherwise. When I ran my test, I forgot to set the 'test' environment mode, so a connection was established to the production database, which was promptly deleted.

I remember exactly where I was sitting, in my apartment next to Dolores Park. The tests ran quickly at that point. So when I started my test and it just hung, I immediately knew something was wrong - but I thought it was a connection issue. "Huh," I thought. "That's weird."

Only when I visited a few seconds later did I realize how bad things were. Nothing worked. As explained in the blog post, we always wiped our test DB before running tests. Turns out, people don't want their data wiped on a Sunday morning for no reason.github.com

Everyone who was around helped out, but @rodjek really saved the day by quickly restoring the DB and punting on the `Events` table. We also immediately locked down production so it wouldn't happen again.

@rodjek My main takeaways:

1. Don't let anything access production except production.
2. Save server upgrades for a weekday, when your team is working and can help you recover from mistakes.
3. Test deleting and restoring data regularly.
4. It can happen to you.

@rodjek Anyway, that's my spooky story. Happy Halloween. 🎃