Thread by @andy23tran

1/ From discovering drugs in record time, detecting tumors with superhuman accuracy or replacing medical scribes, the promise of AI in biology is certainly tremendous. But given the frenzy of activity, it is also seemingly difficult to evaluate what is ready for prime time.🧐

2/ A common question that we get from leaders in biopharma and healthcare, as well as investors and operators, is: “how do I assess a new AI-driven technology and make sure it is worth my time/effort/money?”

3/ This is an important question and in this piece, my @a16z partner @vijaypande and I will provide principles to abide by, point out some common pitfalls, and share how we think about evaluating AI-driven bio technologies.

a16z.com/2021/01/08/eva…

4/ Below are what we view as the 6 key questions to ask when to evaluating a new AI product 👇

5/

*(0)Do you really need AI to solve this problem?*

The very first question is not about the product, but about the problem you want to solve. AI is not a panacea, so start by thinking through whether this problem requires, or would even benefit from, an AI-based approach.

6/

*(1) Is it really AI or is it just marketing hype?*

It's very common to confuse (and sometimes intentionally misuse) the term “AI” when really what’s meant is automated data analysis with pre-programmed software.

7/

*(2) Can the platform actually achieve something differentiated?*

How hard it would be for a third party to replicate this technology -- or improve on it? Is there a moat? Are the datasets unique? Can off-the-shelf packages replicate this easily?

8/

*(3) Is it working? How do you know?*

it is critical to know your metrics for a given application. If you are dealing with a classification problem: AUC, sens, spec, etc. are important. If a complex regression problem, error metrics like R^2 or RMSE are key.

9/

*(4) Is it working... too well? *

You have your AI trained, and it is spitting out .99 AUC! BUT...it quickly falls flat once it is released into the wild. The model may have been "cheating". Training data needs to be representative of and generalizable to real world data.

10/

*(5) Did you run a prospective test, the gold standard of any validation?*

When making a final decision on a given technology, nothing beats a carefully run randomized clinical trial-like prospective test to truly validate the AI platform. That is the holy grail of tests!

11/ As AI continues to seep into every corner of bio, we believe these guiding principles are of paramount importance for both practitioners and business partners alike.

12/ Hopefully, this framework can be a point of entry for those who were once standing on the sidelines, to begin assessing whether a given AI-driven product is worth the investment of time and capital.