Thread by @ChristoPhraser

Our paper on evaluating the first three months of the NHS COVID-19 app is out now as a pre-print. A 🧵 1/n github.com/BDI-pathogens/…

@markbriers @cholmesuk and I have written a blog about the research here

https://twitter.com/markbriers/status/1359061685613584385

The app has been downloaded on over 21 million unique phones, and is used regularly by approximately 16.5 million people (more on this later) 2/n

The app has a privacy-first design, does not track people, and does not share details of contacts or check-ins. We based our data on limited anonymous data based on user-entered postcode areas, and stats such as number of notifications and numbers of positive test results. 3/n

We used two complementary methods to estimate cases and deaths averted during 2020. 4/n

We divided the analysis into multiple phases, reflecting improvements to the function of the app in late October, and which also coincided with the appearance of the "Kent strain" B.1.1.7 5/n

Improvements to the app led to increases in the number of notifications sent out. (The decrease in Nov is due to the lockdown) 6/n

These improvements are reflected in an increase in the inferred effect of the app, which is reassuring in terms of 'sense-checking' the analysis. 7/n

Overall, the effect is a significant reduction in cases for every % increase in app use, with uncertainty about the precise magnitude. 8/n

We sense-checked this further for confounding by reporting placebo analyses and by reporting different methods. The different methods actually gave a larger effect size (% reduction for every % increase in users). 9/n github.com/BDI-pathogens/…

We reported on ways in which the intervention could be improved. The main improvement would, unsurprisingly, be to increase users, but other things can be done too. 10/n

We also looked at the accuracy of the app function itself by estimating the proportion of people who receive a notification who go on to test positive, which is 6.1% 11/n

This is lower than the proportion of notified people who are infected, since not all go on to test positive. Some are asymptomatic, some symptomatic people don't test, and some people don't enter test results in app. 12/n

Comparing this to manual contact tracing was a little tricky. Manual contact tracing results in fewer contacts per case, but from more cases; two thirds are in households. The app sends 4.4 notifications per index case, so logically most must be non-household contacts. 13/n

PHE reports that 7.3% of 'close contacts' identified by manual contact tracing go on to test positive, and 13.5% of 'direct contacts'. Lee et al estimate different % from similar data. 14/n

Overall, it seems that app notifications seem similar accuracy as manual contact tracing, and is good at reaching other app users. The app is having a measurable effect on cases, concentrated in areas of higher uptake. 15/n

Caveats: Not yet peer reviewed. It is not a randomised experiment. We went to lengths to account for confounding, erred towards conservative design choices, and report on alternative approaches. We cross referenced with expectations from the mechanism of action. 16/n

A final note on users versus downloads. Users seems and is more relevant, but downloads is more 'solid data'. There are persistent issues with Android phones in how they are counted, though no indication that this affects function. 17/n

Mismatch is unknown combination of users downloading but not onboarding, reporting issues about usage, and deletes. Based on iOS, probably fewer deletes than people think, but with the privacy design, we don't know. Overall, amongst users, it is working. 18/n

Big team effort on evaluation: @ChrisWymant @LucaFerrettiEvo @OxfordViromics @mishkendall @markbriers @cholmesuk and many non tweeps. Chapeau to the NHS app development team working round the clock. And mostly to all the app users who have contributed to prevention. 19/n

Not a silver bullet, but a useful tool in the armoury. I'll try to answer queries, but please be patient. 20/end