Andy Tran Profile picture
13 Jan, 12 tweets, 3 min read
1/ From discovering drugs in record time, detecting tumors with superhuman accuracy or replacing medical scribes, the promise of AI in biology is certainly tremendous. But given the frenzy of activity, it is also seemingly difficult to evaluate what is ready for prime time.🧐
2/ A common question that we get from leaders in biopharma and healthcare, as well as investors and operators, is: “how do I assess a new AI-driven technology and make sure it is worth my time/effort/money?”
3/ This is an important question and in this piece, my @a16z partner @vijaypande and I will provide principles to abide by, point out some common pitfalls, and share how we think about evaluating AI-driven bio technologies.…
4/ Below are what we view as the 6 key questions to ask when to evaluating a new AI product 👇

*(0)Do you really need AI to solve this problem?*

The very first question is not about the product, but about the problem you want to solve. AI is not a panacea, so start by thinking through whether this problem requires, or would even benefit from, an AI-based approach.

*(1) Is it really AI or is it just marketing hype?*

It's very common to confuse (and sometimes intentionally misuse) the term “AI” when really what’s meant is automated data analysis with pre-programmed software.

*(2) Can the platform actually achieve something differentiated?*

How hard it would be for a third party to replicate this technology -- or improve on it? Is there a moat? Are the datasets unique? Can off-the-shelf packages replicate this easily?

*(3) Is it working? How do you know?*

it is critical to know your metrics for a given application. If you are dealing with a classification problem: AUC, sens, spec, etc. are important. If a complex regression problem, error metrics like R^2 or RMSE are key.

*(4) Is it working... too well? *

You have your AI trained, and it is spitting out .99 AUC! quickly falls flat once it is released into the wild. The model may have been "cheating". Training data needs to be representative of and generalizable to real world data.

*(5) Did you run a prospective test, the gold standard of any validation?*

When making a final decision on a given technology, nothing beats a carefully run randomized clinical trial-like prospective test to truly validate the AI platform. That is the holy grail of tests!
11/ As AI continues to seep into every corner of bio, we believe these guiding principles are of paramount importance for both practitioners and business partners alike.
12/ Hopefully, this framework can be a point of entry for those who were once standing on the sidelines, to begin assessing whether a given AI-driven product is worth the investment of time and capital.

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Andy Tran

Andy Tran Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @andy23tran

13 Jan
1/ Biotech platforms are rich product engines that can generate endless applications, multiple shots on goal.
But...startup resources are finite.

My @a16z partner @JorgeCondeBio and I describe how to find Product-Disease Fit to pick your best shot. 🎯…
2/ Given that there are on the order of 10,000+ known human diseases and every platform is unique, finding that perfect PDF is not simple.

There are 4 key filters in the funnel: (1) platform edge (2) scientific feasibility (3) commercial attractiveness (4) strategic alignment.
3/ (🌟1st step) Find your unique super power, your platform edge. It’s not enough to solve a problem; what problem can you uniquely solve that no one else can?
Dig deep into the science to find it!

This is extremely important and you shouldn't continue until this is defined.
Read 15 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!