The authors found that these apparently impressive ANNs were poorly generalizable (i.e., the performance was much worse on a new validation set compared to the training set).
Compare the red vs. green ROC curves. The performance drops from an AUC of 0.99 to 0.7! Yikes! 2/
There’s a reason for this: They used one dataset for all their positive images and a separate dataset for all their negative images.
This is risky for confounding because the model could pick up on any number of differences in CXRs that aren’t clinically meaningful.
3/
Turns out the neural networks weren't actually looking at the chest in the CXR; they were ‘cheating’ by looking at laterality markers and other parts of the images that were different between the two datasets.
Look at the salience maps (pixels the model deems most important): 4/
The AI ‘learned’ that specific laterality markers used in one hospital go with the COVID19+ cases & a different marker used in a different hospital go with the COVID19- cases
This marker cheating phenomenon was also described in PNA diagnosis in 2018 arxiv.org/abs/1807.00431 5/
3 things we do:
🤔be skeptical: ROC of 0.99 for detecting COVID on CXR is simply too good to be true
💾use better data: use a training set that has both + & - examples, validate with another dataset
👩⚕️medical experts should be involved in developing AI models to spot these biases
• • •
Missing some Tweet in this thread? You can try to
force a refresh
Damn. Under Trump the White House Medical Unit was a pill-mill. Thousands of ambien & provigil per month.
Worse, for a clinic that doesn’t typically do procedures w/ moderate sedation they sure are they ordering prodigious quantities of morphine, fentanyl, versed, & ketamine…?
Honestly, this reminds me of Norman Ohler’s Blitzed.
The AG report was largely concerned with the enormous cost of prescribing these non-genetic meds.
It’s worth pointing out that dispensing prescription meds without documentation is malpractice. In the case of controlled substances it’s also likely a crime.
The long awaited #COVIDOUT RCT is now in @TheLancet:
- high risk adults randomized to either metformin (MET), ivermectin (IVM), fluvoxamine (FLV) or placebo.
- MET reduced the risk of long COVID (6.3% vs 10.4%; NNT = 24)
- no benefit with IVM or FLV
Pulmonary teaching case: you are called to the bedside of a 60yo man who was admitted for pneumonia a week ago. You were called because “he coughed and now his chest is PULSATING!”
This is what you see at the site of a previously removed chest drain:
EN is a rare complication of an infected pleural effusion where purulent fluid “escapes” the pleura and erodes into the chest wall, causing an extrapleural fluid collection that communicates with the pleural space.
Because Empyema necessitans communicates with the pleural space, fluid can move back & forth with respiration, as seen here:
With inspiration, negative intra-thoracic pressure pulls the fluid into the chest. With expiration, positive intra-thoracic pressure pushes fluid out. 3/ twitter.com/i/web/status/1…