Thread by @patio11

Here is what VaccinateCA.com did today on Day 13, to help Californians access reliable information about the coronavirus vaccine so that they can make better informed choices for themselves and anyone they arrange care for:

* We called several hundred pharmacies across the state of California, and wrote down what they said.

* We continued doing QA work with our call center, to align expectations regarding notetaking such that information is displayed on the website in a consistent fashion.

* We briefly published mistaken information. It suggested the coronavirus vaccine was widely available at a pharmacy where it was factually not, resulting in a large number of phone calls to that pharmacy.

We were apprised of this via:

We keep the team staffed to handle data corrections. However, by unfortunate coincidence, this information was up during broadcast coverage of our operations, the ~5 minutes we believe it took to take down the info were high stress for that pharmacy.

We have completed a postmortem on the incident and are instituting a number of controls to prevent similar incidents again.

One of the core principles of reliable systems is that "Simply don't make mistakes" is not a viable strategy. Rather, you have to learn from mistakes.

"Postmortem" is a term of art in the engineering community.

Briefly, it is a formalized process which conducts a root cause analysis regarding the failure of a system to function as designed, identifying ways to improve the technical and human systems implicated.

The engineering community is trending towards a norm of so-called blameless postmortems.

Accountability is an interesting word; some institutions believe it is a synonym for "being found at fault" and therefore exists primarily outside of the building.

Generally speaking, the engineering community uses the term to mean "the state of being a person specifically tasked with ensuring that the opportunities for improvement identified in a system are actually pursued an implemented."

We will follow up tomorrow with the accountable team members to review progress on the remediations they are directly responsible for implementing or seeing implemented.

We also discovered some opportunities for improvement in our incident management process.

For example, we come from a variety of organizations, professional backgrounds, nations, and linguistic traditions. We were delayed during the incident process by failure to have meeting of the minds as to what imprecise words meant with respect to prioritization.

I am the directly responsible individual, having volunteered for it, for writing a standardized tier system to quickly communicate subjective judgments of severity / urgency / etc during an incident, and am accountable for seeing that this is actually written and adopted.

In other improvements news, we're writing down a list of everything which needs to happen on a regular cadence, so that we have committed staffing such that that does factually happen on a regular cadence.

This morning's update, for example, was not posted because I was heavily scheduled, and nobody noticed it until I did prior to the evening update.

That's very low impact, but it clearly isn't sustainable to need a particular person to remember to do something daily for forever.

And so we'll write up a list of tasks like this and make sure they're scheduled.

*Individual people* need to be able to take time off or even have unexpected availability.

The *system*, on the other hand, needs no holidays, knows no weekend, etc.

Because we're in a pandemic.

An an example of cultural differences in engineering, I was informed by colleagues that I surprised them when, during the winddown of the incident response, I said "Statement of intent: I wish to stand down from the incident to travel to a medical appointment. Objections?"

Some of my colleagues report being surprised that I would ask permission to do this, as someone who was not actively involved at that moment in the remediation.

My view on the matter was different, and I didn't realize it until someone pointed it out: I came up in a particular engineering culture in Japan.

In it, an executive leaving an ongoing incident while the engineers could not would be demonstrating catastrophically poor judgement

So the executive followed the standard procedure on having private information regarding human life or safety: share information, clearly state intentions, wait for acknowledgement to act.