Thread by @NoahHaber

Now that everyone is (justifiably) up in arms about CurateScience, may I turn your attention to SciScore™, claimed to be "the best methods review tool for scientific articles." (direct quote, plastered all over their website)

sciscore.com

For starters, for a "tool" all about transparency, reproducibility, etc., it's pretty damn hard to find what it is they are actually reviewing for.

There is no page explaining it, nothing in the FAQ, and you have to dig around a bit.

To find that, there is a link on the "RTI" tab that goes to the preprint:

"Rigor and Transparency Index, a new metric of quality for assessing biological and medical science methods"

doi.org/10.1101/2020.0…

FWIW; having a paper for their metric is a good thing, at least.

So what is it that "the best methods review tool for scientific articles" has selected as its indicators?

This. That's it. That's the whole thing.

This is an extremely shallow and unsophisticated set of metrics that are being inappropriately applied universally.

Now, nothing on this list is inherently a bad thing, but it's not even REMOTELY a meaningful framework for methodological quality or review. It's shallow, weak, vastly overgeneralized, and completely unvalidated against better metrics.

As a "methods review tool" it's nonsense.

One might want to start with a well-vetted framework, thorough discussion of the review methods and ideas before, say, tossing a machine learning algorithm on that framework, producing "reports" and tweeting them out for massive numbers of preprints in the bio/med lit, maybe?

It has all of the same basic, foundational problems as CurateScience's "audits" have, plus more.

On top of how hard to actually see their "tool", it's pretty hard to figure out who makes/runs it. That's really not good for something about transparency and accountability.

Please don't searching for them go piling on, this is not to shame the individuals making it who are likely to be well intended. It's just to point out that the tools we use to rigorously evaluate transparency and rigor must themselves be transparent and rigorous.

There are probably dozens of SciScores and CurateSciences and worse out there that we just haven't seen yet. It's inevitable.

If we can't collectively get our acts together and start investing in impactful and rigorous solutions to these major problems, that's all we'll get.