Thread by @DellAnnaLuca

This report, published 2 months before the COVID-19 outbreak, got it so wrong that it's worth asking ourselves what could avoid similar failures.

More competent people, yes, but there's more. Thread.

https://twitter.com/NightmarePetrol/status/1346200094840934401?s=20

For the curious, the full report is here: ghsindex.org/wp-content/upl…

In the next tweets a few highlights.

The report got so many things wrong. For example, it gave maximum preparedness scores to the US, a country that didn't act like it was very prepared.

Does the fact that the report was largely US funded matter? Did it "force" a high score?

https://twitter.com/DellAnnaLuca/status/1346550622066249730?s=20

I don't think that the numbers have been tortured so that the US come up first.

Perhaps they were, but I believe that the root problem is that people look at their own turf to know what's good, and therefore end up assigning better grades to what looks like what they know.

For example, lots of what was believed to be "normal and common sense" centuries ago are considered today aberrations or idiocies.

A medieval person might see the past as more civilized than current observers see it just because he grew there.

Similarly, it's normal that people who were professionally formed in some academic circles will see more positively what has been influenced by the same circles. For example, US policies and US pandemic preparations.

Generalizing, many statisticians look at n, the number of observations in a study, as a first measure to estimate its validity.

However, real-world statisticians also know that a large n doesn't matter if the observations are all correlated.

Similarly, I don't care how many PhDs there are in a committee if they all received the same education.

Diversity has two benefits. One, inclusion. Two, avoidance of blind spots. A committee can be fully diverse on the first dimension (e.g., comprising different ethnicities) and still be not diverse on the latter (if they all come from the same social / professional circle).

Now, I do not know if the members of the committee that advised on this report were diverse enough; again, this thread is more for general educational purpose rather than to blame this specific report.

That said, I find so many mistakes so baffling to raise a few eyebrows.

For example, it doesn't take medical expertise other than knowing the millennia-old germ theory to know that North Korea third-from-the-bottom in pandemic preparedness doesn't make sense.

They can close their borders as fast and well as few other countries; a clear advantage.

Similarly, I doesn't take medical expertise to ask the question, "shouldn't PPE stock be a factor to include in pandemic preparedness?"

And yet, it doesn't seem that the committee give much importance to it.

I hope that such glaring oversight show the principle that "many eyes and many certifications don't matter if they all come from the same circle."

It's a principle that we should better learn fast, where it matters, to avoid getting blindsided again by the next catastrophe.

Another good advice: don't just study the numbers; study the properties.

Questions such as, "germs spread; how can they get in, how can the country get them out, and how willing it is to do so before it's too late?" are great questions to ask even in absence of quantitative data

Now, this thread didn't even began to scratch the surface of how such kind of manuscripts should be written.

But hopefully it provided two red flags:
– n=1 to the sources of expertise, due to correlation
– only quantitative data, no study of properties

Here’s how some reports work, if they are written by many different experts but all with the same education and social circle.