Thread by @deeksj

21 WAYS TO SPIN RESULTS FROM A COVID DIAGNOSTIC TEST ACCURACY (morning thoughts whilst walking the dog - please add others I've missed). I'm putting this out to help us critique the "number theatre" being thrown at us.
#diagnostics #COVID19 @deb_cohen @MHRAgovuk @drhelmac
1/22

#1 Use a small sample size (and never report how big it isn’t). Use phrases like"we detected all cases" the test is "100% accurate" without mentioning it is only from 29 samples.
2/22

#2 Talk about the total sample size when interpreting results about sensitivity (especially if you have tested 5000 blood donors to estimate specificity and only 29 to estimate sensitivity)
3/22

#3 Don’t quote confidence intervals (if lady luck gave your 29/29 positives - mentioning that we can only be confident that about 80% of patients will be detected doesn't make good press)
4/22

#4 Test people multiple times (but don’t say how many times and when – this magic trick makes the sample look a lot bigger and the results more certain than they are). So if you bled 3 people 7 times, and one 8, you've got 29 samples. Don't mention the only 4 patients bit.
5/22

#5 Select samples to test which are most likely to give strong positives or strong negatives (so prefer samples from those the most seriously ill who have just mounted an antibody respose and recovered, and healthy people who gave a blood donation last year before COVID)
6/22

#6 Select samples which somebody else has already tested that shows that they give strong positives or strong negatives (good guarantee that they will come up trumps). Those trustworthy sample banks in the freezer just keep delivering
7/22

#7 Exclude groups most likely to give false positives or false negatives (don’t ever test people with other respiratory diseases and infectious – although actually that’s really the group we need to know about. Only test the minimum number samples for cross-contamination)
8/22

#8 Only use the samples for which you have large quantities of blood/sera (small samples may be harder to get results from – even though many which we’ll get through the post will be poor)
9/22

#9 Even if the test says it can be used with blood or plasma, use plasma even though most people will test it is blood. If it can be used with fingerprick blood, stick with the stuff from veins (less like to go wrong)
10/22

#10 Use the most professional and expert staff possible to do tests (reduces the chance of human operator errors - which will occur when a test is implemented in the NHS)
11/22

#11 Use the most highly resourced laboratories to do the tests (no equipment failures or contamination risks - which will occur when a test is implemented in the NHS)
12/22

#12 Delete outliers and cases from data which are different (not including people who have reduced immune responses in the samples will make it look better - even though they are real people and will be tested like everybody else)
13/22

#13 Never report on points (4)-(12) (the tradition of only writing two sentences about study methods in the Instructions for Use sheet means this never happens and nobody will ever know).
14/22

#14 Find subgroups in whom the test works well and focus on reporting those results (like when my family shares a cake, there will always be one bit better than the others). If possible never mention the other groups.
15/22

#15 Don’t register the study and publish a protocol before starting (others would be able to check what changes the investigators have made during the study which might make the results look better)
16/22

#16 Don’t publish a statistical analysis plan before analysing the data (others would be able to check what changes the investigators have made during the data analysis and reporting which might make the results look better)
17/22

#17 Don’t publish if the results aren't good, or publish without naming the test, or only publish results which give the highest estimates. Use your legal team to ensure that non-disclosure agreements make it impossible for results to be reported if they turn out bad.
18/22

#18 Don’t make the dataset freely available (would allow independent verification of results)
19/22

#19 Publish your results in a newspaper first (all criticism of the study by scientists will be old news and sour grapes by the time they get a chance to make it, and government policy will already have been made).
20/22

#20 Only link your website to the study report in which you test looked best, and do your best to distract from the rest. Never contribute to meta-analyses where all the good and bad will be published.
21/22

#21 Never compute predictive values (even though we neeed them to show the true value of positive and negative test results for patients and decision makers, they won't be as impressive).
22/22

@threadreaderapp unroll