Thread by @patio11

A long time ago, I worked for a Japanese company which had a Alert Lots Of People function available to it. We drilled on its use.

Since it's instructive (this thread may be longish):

A: "I am declaring [an emergency / a drill] for [area]. Get the book."
B: "I am secondary on [an emergency / a drill] for [area]. Here is the book."

A flips to Page 1 and reads down a checklist in order.
A: "I am attempting to log into [production / test] environment."

B: "Confirming you are attempting to log into [production / test environment]."

A: "Login complete."
B: "I verify you are logged into production. Proceed."

A: "I am queueing up an alert, entitled $TEXT."
B: "Confirm to me that your expected text is [reads screen]."
A: "Confirm"

A: "I am sending this alert to $GROUP."
B: "I acknowledge you will send the alert to $GROUP."
A: "I have selected $GROUP in the interface. Confirm."
B: "Confirmed."

A: "I will now send the alert."
B: "You are good to send the alert."
A: "I have sent the alert."

They then verify that the alert was in fact sent and verify it was received on one of the canaries, using a similar level of pedanticness.

This could be circumvented by any engineer, on their own authority, if there was a danger to human life. (e.g. "I became aware of a tsunami at 3 AM in the morning and could not raise a second within 15 seconds.")

We thought about this process *a lot*, and drilled it.

You can file this under "an anecdote from an engineering culture which does a few things really, really well"; this is one of them.

"Was it an actual, physical book?"

Well, technically, a binder, in case we ever needed to update procedures.

"Why not have it on an electronic system?"

Because binders have 100% uptime even if office/remote servers/etc are presently rattling in the wake of e.g. earthquake.

"What happens if the Internet connection was down between you and the system?"

This is a question that gets perilously close to that I Can't Tell You territory, but web developers who have ever coded "Fire X if we don't get a heartbeat from Y" can probably guess.

The above is offered in the spirit of improving plans at your current/future workplaces and not as even an implied criticism of any other systems.

When it really counts, there are only two teams: humanity and disaster. We’re all in it together on Team Humanity.