Thread by @drzhelnov

It’s been two years since I started working on my Reproducible Systematic Subset query for #PubMed to power my knowledge synthesis appraisals at zheln.com, and look what I have found so far ⬇ 1/8

First and foremost, your regular PubMed searches you see when reading through evidence syntheses are fundamentally irreproducible ­– in part, at best. This is because they usually do not account for indexing dates – the issue I fixed in the RSSB 2/8

RSSB, being short for Reproducible Systematic Subset, is the same as PubMed’s regular Syst Rev filter (with both the older and newer versions combined) but tweaked for reproducibility: osf.io/z3ju7/ I.e., it gives you the same record set whenever you rerun the query 3/8

After testing the RSSB for two years, I can tell that the record sets are still reproducible: github.com/p1m-ortho/qs-g… However, several issues arose on PubMed’s side that are worth mentioning 4/8

As I noticed back in 2020, PubMed indexing would fail on some dates, and then the NCBI team would add those records *retrospectively*:

https://twitter.com/drzhelnov/status/1318530167157948417

In other words, records that were not there when one ran the original query may randomly blend in upon rerunning 5/8

My brand new finding at debug is that the PubMed team *deletes* records they mark as duplicate, so they disappear from all the query results. This means that when rerunning a query from an older SR, some records may be silently missing: github.com/p1m-ortho/qs-g… 6/8

All in all, it appears to me that we in evidence synthesis are only starting to foster true reproducibility of our methods – huge props to @mjpages for their groundbreaking #REPRISE project & to @OSFramework for their fundamental repeatability research, among others 7/8

As to the RSSB, I will post more tests on its GitHub page: github.com/p1m-ortho/qs-g… For updates on #Zheln and my other research, stay tuned by following me @drzhelnov Imma peck it to the core ⛏ 8/8