Thread by @GeorgeWHerbert

@QuinnyPig Reinterating my story from Hurricane Sandy.

Was consulting at a small internet service company when Sandy hit. First projected landfall was right into DC / Northern Virginia. Projected eyewall track over all the us-east-1 datacenters I think. Our stuff was all there. 1/

@QuinnyPig 2/ So we quickly stand up configuration management server in AWS Oregon. Ok. Can rebuild site from private Github + CM; just need DNS switch. Done.

Had horrible feeling, called GitHub. “Are you by any chance in us-east-1?”
“Yes.”
“Only us-east-1?”
“Yes, why do you ask?”

@QuinnyPig 3/ We tell GitHub to look at the weather report and copy their infrastructure backups out of the projected storm track. Much hilarity ensues.

We rapidly download whole repo structure to every admins laptop. Ok.

@QuinnyPig 4/ All better. Wait. Failover also requires DNS change. Look up our DNS provider in AWS netblocks list. Ping. Call DNS provider.

“Are you in us-east-1?”
“Yes.”
“Only in us-east-1?”
“Yes why?”

We explain. Poignant pause. “Our office is in Delaware, too.”

Much hilarity ensues.

@QuinnyPig 5/ So they start DR prep, we add a slave nameserver in Oregon & add that to our domain NS records at registrar… who we call and they aren’t *just* in us-east-1 (“Why do you ask?…” … much hilarity ensues).

Fortunately for everyone but New York City hurricane drifts north.