Can you detect COVID-19 using Machine Learning? ๐ค
You have an X-ray or CT scan and the task is to detect if the patient has COVID-19 or not. Sounds doable, right?
None of the 415 ML papers published on the subject in 2020 was usable. Not a single one!
Let's see why ๐
You have an X-ray or CT scan and the task is to detect if the patient has COVID-19 or not. Sounds doable, right?
None of the 415 ML papers published on the subject in 2020 was usable. Not a single one!
Let's see why ๐
Researchers from Cambridge took all papers on the topic published from January to October 2020.
โช๏ธ 2212 papers
โช๏ธ 415 after initial screening
โช๏ธ 62 chosen for detailed analysis
โช๏ธ 0 with potential for clinical use
healthcare-in-europe.com/en/news/machinโฆ
There are important lessons here ๐
โช๏ธ 2212 papers
โช๏ธ 415 after initial screening
โช๏ธ 62 chosen for detailed analysis
โช๏ธ 0 with potential for clinical use
healthcare-in-europe.com/en/news/machinโฆ
There are important lessons here ๐
Small datasets ๐
Getting medical data is hard, because of privacy concerns, and at the beginning of the pandemic, there was just not much data in general.
Many papers were using very small datasets often collected from a single hospital - not enough for real evaluation.
๐
Getting medical data is hard, because of privacy concerns, and at the beginning of the pandemic, there was just not much data in general.
Many papers were using very small datasets often collected from a single hospital - not enough for real evaluation.
๐
Biased datasets ๐ง๐งโ๐ฆฒ
Some papers used a dataset that contained non-COVID images from children and COVID images from adults. These methods probably learned to distinguish children from adults... ๐คทโโ๏ธ
๐
Some papers used a dataset that contained non-COVID images from children and COVID images from adults. These methods probably learned to distinguish children from adults... ๐คทโโ๏ธ
๐
Training and testing on the same data โ
OK, you just never do that! Never!
๐
OK, you just never do that! Never!
๐
Unbalanced datasets โ๏ธ
There are much more non-COVID scans than real COVID cases, but not all papers managed to adequately balance their dataset to account for that.
Check out this thread for more details on how to deal with imbalanced data:
๐
There are much more non-COVID scans than real COVID cases, but not all papers managed to adequately balance their dataset to account for that.
Check out this thread for more details on how to deal with imbalanced data:
https://twitter.com/haltakov/status/1359643910688157696
๐
Unclear evaluation methodology โ๏ธ
Many papers failed to disclose the amount of data they were tested or important aspects of how their models work leading to poor reproducibility and biased results.
๐
Many papers failed to disclose the amount of data they were tested or important aspects of how their models work leading to poor reproducibility and biased results.
๐
The problem is in the data ๐ฝ
The big problem for most methods was the availability of high-quality data and a deep understanding of the problem - many papers didn't even consult with radiologists.
๐
The big problem for most methods was the availability of high-quality data and a deep understanding of the problem - many papers didn't even consult with radiologists.
๐
A high-quality and diverse dataset is more important than your fancy model!
References ๐๏ธ
Full article in Nature: nature.com/articles/s4225โฆ
More detailed coverage: statnews.com/2021/06/02/macโฆ
Source for X-Ray image: bmj.com/content/370/bmโฆ
Full article in Nature: nature.com/articles/s4225โฆ
More detailed coverage: statnews.com/2021/06/02/macโฆ
Source for X-Ray image: bmj.com/content/370/bmโฆ
I agree with all your points! I'm sure the data basis is much better now and I've seen much larger datasets.
I think the important point is that rushing to publish results based on small and bad quality datasets undermines the credibility of ML...
I think the important point is that rushing to publish results based on small and bad quality datasets undermines the credibility of ML...
https://twitter.com/BulletproofLS/status/1402757293892988930?s=20
There are already much better datasets and I'm sure this problem will get solved at some point. There is also a promising Kaggle challenge:
kaggle.com/c/siim-covid19โฆ
The point is that it needs to be done right and the Nature paper gives some guidance how.
kaggle.com/c/siim-covid19โฆ
The point is that it needs to be done right and the Nature paper gives some guidance how.
https://twitter.com/aruj_budhraja/status/1402762164390166531?s=20
Yes, this is a common problem with such datasets/competitions. At some point people start figuring out how to fine tune on the test set...
One should keep adding data to the test set!
One should keep adding data to the test set!
https://twitter.com/SzassTam/status/1402768264363655168?s=20
There is certainly pressure to publish in academia
I think many of the papers really wanted to help with getting results out quickly. However, there are some minimum standards that need to be kept. Such results undermine the credibility of ML...
I think many of the papers really wanted to help with getting results out quickly. However, there are some minimum standards that need to be kept. Such results undermine the credibility of ML...
https://twitter.com/AmariMatthews10/status/1402780210668314625
They analyzed papers using both CXR and CT. I would argue that you first need to fix the dataset and then optimize your model. You can usually get much better results addressing things in this order.
https://twitter.com/VidurMahajan1/status/1402811611971133450
An interesting point for the people asking what is the benefit of detecting COVID-19 in X-ray scans at all.
https://twitter.com/ChuckDeo/status/1403071863794724864?s=20
โข โข โข
Missing some Tweet in this thread? You can try to
force a refresh
ใ
