Can you detect COVID-19 using Machine Learning? π€
You have an X-ray or CT scan and the task is to detect if the patient has COVID-19 or not. Sounds doable, right?
None of the 415 ML papers published on the subject in 2020 was usable. Not a single one!
Let's see why π
You have an X-ray or CT scan and the task is to detect if the patient has COVID-19 or not. Sounds doable, right?
None of the 415 ML papers published on the subject in 2020 was usable. Not a single one!
Let's see why π
Researchers from Cambridge took all papers on the topic published from January to October 2020.
βͺοΈ 2212 papers
βͺοΈ 415 after initial screening
βͺοΈ 62 chosen for detailed analysis
βͺοΈ 0 with potential for clinical use
healthcare-in-europe.com/en/news/machinβ¦
There are important lessons here π
βͺοΈ 2212 papers
βͺοΈ 415 after initial screening
βͺοΈ 62 chosen for detailed analysis
βͺοΈ 0 with potential for clinical use
healthcare-in-europe.com/en/news/machinβ¦
There are important lessons here π
Small datasets π
Getting medical data is hard, because of privacy concerns, and at the beginning of the pandemic, there was just not much data in general.
Many papers were using very small datasets often collected from a single hospital - not enough for real evaluation.
π
Getting medical data is hard, because of privacy concerns, and at the beginning of the pandemic, there was just not much data in general.
Many papers were using very small datasets often collected from a single hospital - not enough for real evaluation.
π
Biased datasets π§π§βπ¦²
Some papers used a dataset that contained non-COVID images from children and COVID images from adults. These methods probably learned to distinguish children from adults... π€·ββοΈ
π
Some papers used a dataset that contained non-COVID images from children and COVID images from adults. These methods probably learned to distinguish children from adults... π€·ββοΈ
π
Training and testing on the same data β
OK, you just never do that! Never!
π
OK, you just never do that! Never!
π
Unbalanced datasets βοΈ
There are much more non-COVID scans than real COVID cases, but not all papers managed to adequately balance their dataset to account for that.
Check out this thread for more details on how to deal with imbalanced data:
π
There are much more non-COVID scans than real COVID cases, but not all papers managed to adequately balance their dataset to account for that.
Check out this thread for more details on how to deal with imbalanced data:
https://twitter.com/haltakov/status/1359643910688157696
π
Unclear evaluation methodology βοΈ
Many papers failed to disclose the amount of data they were tested or important aspects of how their models work leading to poor reproducibility and biased results.
π
Many papers failed to disclose the amount of data they were tested or important aspects of how their models work leading to poor reproducibility and biased results.
π
The problem is in the data π½
The big problem for most methods was the availability of high-quality data and a deep understanding of the problem - many papers didn't even consult with radiologists.
A high-quality and diverse dataset is more important than your fancy model!
π
The big problem for most methods was the availability of high-quality data and a deep understanding of the problem - many papers didn't even consult with radiologists.
A high-quality and diverse dataset is more important than your fancy model!
π
References ποΈ
Full article in Nature: nature.com/articles/s4225β¦
More detailed coverage: statnews.com/2021/06/02/macβ¦
Source for X-Ray image: bmj.com/content/370/bmβ¦
Full article in Nature: nature.com/articles/s4225β¦
More detailed coverage: statnews.com/2021/06/02/macβ¦
Source for X-Ray image: bmj.com/content/370/bmβ¦
This week I'm reposting some of my best threads from the past months, so I can focus on creating my machine learning course.
Next week I'm back with some new content on machine learning and web3, so make sure you follow me @haltakov.
Next week I'm back with some new content on machine learning and web3, so make sure you follow me @haltakov.
β’ β’ β’
Missing some Tweet in this thread? You can try to
force a refresh
γ
