Thread by @SarahDRasmussen

ONS, breaches, + teachers: THREAD

Today I reported an ONS breach to the UK Statistics Authority.

Two breaches in fact: one regarding an “ad hoc” analysis on education staff in part 9 of the 6 Nov ONS covid survey pilot, and one for a related 3 Nov government leak. 1/

On 3 Nov, the Department for Education tweeted a video of Chris Whitty saying:

“All the data, including ONS data, do not imply that teachers are a high risk occupation, unlike, for example social care workers, and medical staff like myself.” 2/

https://twitter.com/educationgovuk/status/1323701607159099392

According to the UK Statistics Office “Code of Practice for Statistics,”
code.statisticsauthority.gov.uk/wp-content/upl…

this was a breach of trustworthiness principle T3: Orderly release. 3/

T3.4 states that “ahead of their publication [...] no indication of the statistics should be made public.”

T3.8 says “Policy, press or ministerial statements referring to regular or ad hoc official statistics should [...] contain a prominent link to the source statistics.” 4/

This 3 Nov announcement led to confusion, as before 6 Nov, the only published occupation-differentiated ONS covid data was on deaths during spring lockdown.

And although Whitty said “all the data, including ONS,” he never ended up identifying non-ONS data on this topic. 5/

The more important breach, though, has to do with the 6 Nov ad hoc analysis itself, which defied the “Code of Practice for Statistics” in so many ways as to raise serious concerns about independence. 6/
ons.gov.uk/peoplepopulati…

Though the other analyses in this 6 Nov report used data through *31 Oct* & included time dependence, the ad hoc teacher analysis only used 2 Sep-*16 Oct* data, & suppressed time dependence.

This matters: 17 Oct is about when 2-11-yr-olds overtook 35-49-yr-olds in infection. 7/

Why omit the 17-31 Oct occupation data—especially when this omission was so likely to bias results?

Did they feel they had run out of time to analyse already-acquired data? Code of Practice says that’s an unacceptable reason, especially considering the 3-day-early leak. 8/

OTOH, with teacher training, many schools started Mon 7 Sep instead of Thu 2 Sep. Infection takes time to incubate before testing +.

So up to 10 days worth of their early data predated the option for school-acquired infection, diluting total positivity results for teachers. 9/

Wait, it gets worse: just LOOK at the error bars!

Data was disaggregated in such a way as to make the study totally underpowered.

Error bar width is
>2 times the positivity value for most edu categories,
>33% higher than positivity for ALL BUT ONE edu category.

10/

But there was one exception: “teacher of unknown type,” with error bar 35% SMALLER than the positivity—meaning most respondents simply answered “teacher.”

THIS teacher category had
28% higher positivity than key workers,
16% higher positivity than other professions. 11/

So really, the chart should have just looked like the one below, with more sensible error bars,

and with teachers in the highest positivity category by far, even without the 17-31 Oct data.
12/

It was only the disaggregation of teachers into underpowered buckets that made it plausible for ONS to claim:

“Data [...] show no evidence of differences in the positivity rate...” in the text and “there is no evidence of difference in the positivity rate” in the caption. 13/

These “no evidence of differences” comments don’t mean what the audience is intended to think they mean.

Rather, they mean, “we disaggregated and restricted the data in such a way as to make the analysis too underpowered to generate significant evidence.” 14/

This badly violates the Code for Practice.

And saying “no evidence of difference” vs “evidence of no difference” isn’t a get-out-jail-free card.

Context matters. Readers expect that if you publish an ONS analysis, the verdict is not: this was underpowered by design. 15/

The ONS are also obligated to provide rationale for substantial deviations from standard methodology, such as why they omitted 17-31 Oct data and suppressed time dependence for this one analysis, in a report that consistently provided time-dependent data through 31 October. 16/

On a side note, it contradicts Swedish findings.

In spring, Swedish teachers of upper sec schools (fully remote) had median infection rates.

Teachers of (fully open) lower sec schools had nearly twice the % of confirmed cases, and were among the most infected professions. 17/

It is not the role of the ONS to manufacture false evidence for purposes of state propaganda. This message of this one analysis was

-announced by Chris Whitty on 3 Nov,
-tweeted by the Department for Education on 3 Nov, ...
18/

https://twitter.com/educationgovuk/status/1323701607159099392

-invoked by SAGE in the 4 Nov (published 13 Nov) report “TFC: Children and transmission 4 Nov, 2020,” where it was the only included “evidence” on teacher risk—in fact they (predictably) changed “no evidence of difference” to “show no difference”,...
gov.uk/government/pub…
19/

-tweeted on 13 Nov by the Department for Education (this time by invoking the 4 Nov SAGE report) as new evidence of school safety, even though much of the actual child-related data in the report involved strong evidence of substantial transmission,...
20/

https://twitter.com/educationgovuk/status/1327246747102552066

-and invoked 18 Nov by Scottish Government as part of their evidence of school safety from covid,...
gov.scot/news/coronavir…
gov.scot/publications/c…

(Again, “no evidence of difference” became “found there is no difference.”) 21/

As an American-born, this all brings back too many bad memories of George W Bush’s “weapon’s of mass destruction.”

If you push hard enough for the manufacture of evidence, evidence will be produced, whether it’s valid or not.
22/

That’s why ONS independence is so incredibly important.

We can’t let this happen again.

23/end

Since this kind of blew up, I wrote a follow-up thread. Wanted to clarify several things. Also, sorry for instances of inflammatory language in the above thread. Trying to be a good Twitter communicator, but still learning. 24/

https://twitter.com/SarahDRasmussen/status/1330744740711698433