Thread by @Nightingale_P

Weekend longread for the #sciencepolicy crowd.

@AnEmergentI and I have been digging at how good UK research really is. And have come up with some worrying findings....

S&T - Is the UK a world leader in science? jameswphillips.substack.com/p/s-and-t-is-t…

There is a lot of guff about how world leading UK science is, and James and I have worked in US, Switzerland, Korea, Denmark etc.... and shared a little scepticism.

So we asked "Is it *really* though?"

Some of the "evidence" can be dealt with quickly.

1. Commercial university rankings.... ha ha ha!

2. REF results. At least this is based on some evidence. We quote the Russell group on how it allegedly shows 94% of their research is world leading etc... Total guff.

3. REF only looks at UK research (you'd be pretty clever to jump to international comparisons) & isn't a random sample. Unis select their best outputs & submit them. This selection bias means you can't generalise anymore than I can about my fitness from UK Olympic medal scores

4. Nobel prizes are backwards looking and don't tell us much about the quality to research today.

We also have citation scores... I could write a book about how careful you need to be when using these.

See for example this paper on how their biases

arxiv.org/abs/1105.1227

5. The UK Govt has done nice data that shows excellent UK performance on the most highly cited papers. USA and China aren't on this pic because they are off the scale.

Big questions about if you should normalise for population GDP etc. But let's run with it...

6. One thing you should do is normalise for fields of research because they have different citation patterns. If you don't you can get strange results - I have one published ranking that puts Sussex ahead of LSE in economics! We are good. But we ain't that good.

7. When you do that. The UK is still good. But the lead declines substantially. We are about a good as Germany & Italy. That's v good but shoes the "13% of most highly cited research" quote is misleading.

This is really good performance. But it's not science superpower.

8. I was making the usual comments - rankings are misleading, research has a direction but just a rate, research systems can be v badly out of sync with societal and economic needs etc...

And James made a really insightful point.

Which I'll try and explain.

9. He pointed out that being in the top 1% sounds good, but science is *so* big that the top 1% will contain thousands and thousands of papers, many of which won't by definition be the very best.
....

10. And he pointed out the UK research system is v efficient and respond to incentives very well. But the incentives in REF (see caveats before) are for 4* papers. That means judged by UK academics to be internationally good.

11. So there is no extra incentive to go the extra mile to produce the really outstanding cutting edge research that is game changing.

Arguably you are incentivised *not* to do that b/c it takes time and money.

12. This is important because there is evidence extremes drive science. And there are huge first mover advantages in many areas of tech.

So rather than the metrics showing UK science is excellent they made 1. Be hiding a problem, and 2. Be part of the cause of the problem.

13. This then raises the question - how does the UK perform at the absolute cutting edge of research?

We couldn't look at all research areas, so we looked at the three strategic areas in the Integrated Review - AI, Quantum and SynBio.

14. Twitter is glitching...

What we find was concerning.

Please have a look.

jameswphillips.substack.com/p/s-and-t-is-t…

15. The method are "rough and ready" and the data isn't great. But we do try and triangulate across evidence types and present the most roses evidence.

We'd really appreciate comments.

16. As an aside - If I was the VC or ProVC for Research at Manchester, Warwick, Bristol or Edinburgh I would ask the Government to use more sophisticated metrics.

And I would strongly resist this if I were at....