Thread by @venkmurthy

More hype around AI, mHealth, wearables. @AliveCor devices can use #AI to find LQTS:

prnewswire.com/news-releases/…

From the press release:

AUC 0.83, specificity 81%, sensitivity 73%, accuracy 79%

Prevalence of LQTS 160,000 out of US population of 325.7 million (0.049%)

Do we need to do a tweetatorial about how this is grossly insufficient accuracy and will almost certainly lead to more harm than good?

Let's say @GundotraVic is wildly successful and sells these things to 10% of the US population... how many is that?

If this was a random sampling of 10% of the US population, how many of the 160,000 LQTS patients would have these devices:

Now lets say its not a random sampling but enriched for LQTS at 2X the rate of the population. How many people have LQTS and one of these devices?

I'm going to wait here for a bit for people to answer these questions.

@onco_cardiology @purviparwani @ProfDFrancis @ProfHayward @JeremySussman @bnallamo @mdmajmudar @RogueRad @SVRaoMD @ADAlthousePhD @barttels2 @VinayPrasadMD @AnilMakam @thebyrdlab @raj_mehta @JoaoLCavalcante

Okay. About ten votes so far. Hopefully more soon.

Let's take the most popular answers thus far.

~32 million people with devices
~32 thousand of them with LQTS

So what fraction of @AliveCor users have LQTS?

Given ~0.05% of the US population has LQTS and we are guessing 2X enrichment in the @AliveCor user group, thr above question should be easy!

Next questions are hard. A few abbreviations first.

TP = True Positive, meaning pt has LQTS and test is abnl

TN = True Negative, meaning pt does not have LQTS and test is normal

FP = False Positive, meaning pt does not have LQTS but test is abnl

FN = False Negative, meaning pt has LQTS but test is normal

please forgive me for the sins I am about to commit.

Sensitivity is the True Positive Rate or TP / All Disease Positives. All Disease Positives are TP + FN. So Sensitivity = TP / (TP + FN)

Specificity is the True Negative Rate or TN / All Disease Negatives. All Disease Negatives are TN + FP. So Specificity = TN / (TN + FP)

So which measure is more useful to calculate your false positive rate?

Good! Now which measure is most useful for calculating the false negative rate?

Once I get a few answers to these questions, we'll move on to the next part...

Okay, for the next part, let's round off a few of the numbers thus far:

Total with device: 30 million
Total with device and with LQTS: 30 thousand
Sensitivity: 80%
Specificity: 75%

I'm only doing this to make brain math easier.

What is the number with a device but WITHOUT LQTS?

While we wait for answers to the prior question to roll in, let's turn to sensitivity.

I do some algebra in the picture to show that 1-sensitivity = FN/(TP+FN) = FN/(all pts with disease)

Feel free to ignore the picture in the tweet above if algebra gives you the heebie jeebies, but I think math is fun. This is pretty simple algebra anyway so...

So we miss (1-0.75)=25% of all patients who really have LQTS.

That is 30,000 * 25% = what?

Now lets turn to the other side of the problem. The false positives. I again do some algebra in the attached picture to show that:

1 - Specificity = FP / (TN + FP) = FP / # people without disease

So if we want to know what fraction of people without LQTS will have a false positive result, we just take 1- Specificity = 1- 0.80 = 20%

Doesn't sound so bad... except when you think that this is out of 30 MILLION people without LQTS. so the number of false positives is

Don't read the next tweet until you answer all of the above questions.

By using this device on 30 million people, @GundotraVic is very happy that he is a business success. We find a total of 22,500 cases of LQTS (and miss 7,500). We also falsely label 6 MILLION people with LQTS.

So what fraction of people who get diagnosed with LQTS really have it? This is known as the Positive Predictive Value (PPV).

The formula for PPV = TP / (TP + FP).

In this case PPV = 22,500 / (22,500 + 6 million)

What is that?

Again, don't continue until you have answered all the questions above. No cheating!

The PPV is important because it helps you understand how many people could be harmed by this technology.

For LQTS, the PPV is <1%. So as it turns out 99% of people labeled by this SUPER INNOVATIVE technology will have nothing wrong with them.

They will have doctor bills, genetic testing, possibly changes to their medications with zero potential for benefit.

Is that a good deal?

Here is a very polite tweet from the CEO of @AliveCor

Turns out they aren't anywhere near shipping this as a product. Which is good. But they are using this to drum up interest and maybe to keep investors interested.

Problem is that for rare diseases, you need specificity of 99% or greater to not suffer from a flood of false positives.

It's really nice to say this is just promising, preliminary work but is there really hope to go from 0.80 specificity to 0.999 specificity just by tweaking the algorithm.

I suspect not.

Algorithms and sensors passively identifying rare health conditions is super hard.

An honest evaluation of this might have been "We tried but it turns out its nowhere near fit for widespread screening. We'll think further about other things we could do".

Instead we hear the word "Stunning" to describe the tech because of all the #AI & #MachineLearning hype.

Here is an article in the non-medical press which is takes part in the hype in @engadget:

engadget.com/2018/05/11/ai-…

Don't read @engadget ? How about @CNBC?

cnbc.com/2018/05/10/ali…

“There can be no better illustration of the importance of our AI to medical science than using it to detect that which is otherwise invisible,” said AliveCor CEO Vic Gundotra.

fiercebiotech.com/medtech/mayo-c…

Does this AI have "importance to medical science"?

From the @CNBC article above...

Is this true that LQTS is "often" the cause of sudden death during a game? I always thought about HCM, ARVC, other cardiomyopathy, coronary anomalies, etc were more common.

Most recent data from NCAA athletes says that LongQT causes about 1% of sudden cardiac death.

circ.ahajournals.org/content/132/1/…

LongQT is that tiny slice just before the 12 o'clock position...

Turns out the biggest cause of death of NCAA athletes over all is accidents.

So 99% wrong on positive calls, but let's still screen all high school athletes for something that is maybe responsible for 1% of a rare event (sudden cardiac death) that accounts for a small portion of all deaths of student athletes.

**But think of the children**

I'd rather not harm children by telling them they can't play sports when nothing is wrong with them...

The infamous @doctorwes (of MOC fame) has written a terrific and funny blog post about the harms of excess EKG screening in athletes:

drwes.blogspot.com/2012/05/dark-s…

H/t @EJSMD