Thread by @patio11

One of the more important things to happen for VaccinateCA.com was @jain_ankit (CEO of Infinitus.ai) dropping by, very early in the project, asking how they could help.

They generously donated the use of Eva, their digital assistant. She is *magic* for us.

Eva is, effectively, Amazon Alexa but capable of pushing buttons on a telephone and following simple instructions.

We ask her to go through a long list of pharmacies and ask a simple screening question: "Do you have the vaccine?"

Professionals follow up on all the Yeses.

We are acutely constrained by the number of phone calls we can place per day. Medical providers are acutely constrained right now by their own time in answering phones (trading off with patient care).

Eva saves us from having to make thousands of futile calls.

Instead of making those futile calls to places which either don't have the vaccine or will never have the vaccine, our professionals can spend their time digging into the details for places that do, or debugging issues in the supply chain.

An example of debugging: when we heard a particular pharmacy in a particular county had a way to make appointments but just wasn't getting any ("Hmm, how odd"), a volunteer pulling at that string eventually found the *entire chain* had been left off the appointment booking site.

Fixing that bug brought 11 pharmacies, each with a box of perfectly good unused vaccine sitting in the freezer, online, for the price of a few minutes of concentrated thought and calling around.

Eva buys our volunteers a *lot* of time to be impactful like that.

Not just anecdotally, either. I can quote the results from early in our experiment from memory.

P(vaccine): 12% <-- means "a 65 year old could get vaccinated, today"
P(vaccine | no report from Eva yet): 11%
P(vaccine | Eva called and believes there is vaccine at location): 40%

Me to data science team: "Somebody want to run a chi-squared test on this for me so I can get a p value or can I just say 'statistically significant by eyeball'"

(Sample size, again, in thousands.)

We've got three legs of the calling stool:

* our volunteers, who are embedded directly with the rest of the team and who give us an iteration cycle of ~5 minutes on scripts/targeting/prioritization
* our call center colleagues, who scale higher/faster but on daily latency
* Eva

We're probably going to be able to do even more interesting things with Eva over time, like using her for follow-up calls.

"It's important that we keep this information up to date. Would a daily call be OK with you?" "Sure!" "It's more efficient for us and you if Eva does it."

And then explain what Eva is, etc etc, and that if they could please tell colleagues that if Eva calls on behalf of VaccinateCA that is expected and to just give her the update and hang up.

We'd have one of our professionals call back soon if anything had changed.

"Are pharmacists OK with these calls?"

You'd be surprised how often we hear "I actually don't have any vaccine yet but you should go to this website to find it. Vaccinate, like the verb, CA, like California, dot com. Did you get that?" (More than 5X now.)

(As you'd expect, we respect medical professionals' preferences when they communicate them to us.

Overwhelmingly, they're in favor of telling people about the vaccine; that's why the model works. The person authorized to give the shot knows inventory and policy, ~100% of time.)

As long as I'm on the subject: one reason we would indefinitely take a number out of the list is if they'll never get the vaccine. Internally, this is referred to as "Sir, this is an Arby's."

Two such events: we called a hospital and got told "This hospital is *for horses.*"