Thread by @nealhaddaway

In this paper in @NatureEcoEvo we outline 8 major problems that can occur with traditional ways of reviewing the literature, and provide concrete advice on how to avoid them: rdcu.be/b8pp0

This thread outlines our key points! (1/23)
#EvidenceSynthesis #SystematicReview

Traditional ways of reviewing the literature may be susceptible to bias and end up giving us incorrect conclusions.

This is of particular concern when reviews address key policy- and practice- relevant questions. (2/23)

Systematic reviews aim to maximise rigour and minimise susceptibility to bias using rigorous methods.

But despite these methods being available, poor reviews are still published: these could be improved with a few key processes; some not prohibitively costly. (3/23)

Here follow 8 problems with traditional literature reviews, each with suggested solutions. (4/23)

Problem 1: Lack of relevance - limited stakeholder engagement can produce a review that is of limited practical use to decision-makers (5/23)

Solution 1: Stakeholders can be identified, mapped and contacted for feedback and inclusion without the need for extensive budgets - check out best-practice guidance: stakeholdersandsynthesis.github.io (6/23)

Problem 2: Mission creep - reviews that don't publish their methods in an a priori protocol can suffer from shifting goals and inclusion criteria. (7/23)

Solution 2: carefully design and publish an a priori protocol that outlines planned methods for searching, screening, data extraction, critical appraisal and synthesis in detail. Make use of existing organisations to support you (e.g. @EnvEvidence; environmentalevidence.org/guidelines/sec…) (8/23)

Problem 3: A lack of transparency/replicability in the review methods may mean that the review cannot be replicated (a central tenet of the scientific method!)
onlinelibrary.wiley.com/doi/10.1002/ec… (9/23)

Solution 3: Be explicit, and make use of high-quality guidance and standards for review conduct (e.g. CEE Guidance, environmentalevidence.org/information-fo…) and reporting (PRISMA, prisma-statement.org or ROSES, roses-reporting.com). (10/23)

Problem 4: Selection bias (where included studies are not representative of the evidence base) and a lack of comprehensiveness (an inappropriate search method) can mean that reviews end up with the wrong evidence for the question at hand. bmcmedresmethodol.biomedcentral.com/articles/10.11… (11/23)

Solution 4: Carefully design a search strategy with an info specialist; trial the search strategy (against a benchmark list); use multiple bibliographic databases/languages/sources of grey literature; publish search methods in an a priori protocol for peer-review. (12/23)

Problem 5: The exclusion of grey literature and failure to test for evidence of publication bias can result in incorrect or misleading conclusions. onlinelibrary.wiley.com/doi/abs/10.100… (13/23)

Solution 5: Include attempts to find grey literature, including both 'file-drawer' (unpublished academic) research and organisational reports.

Test for possible evidence of publication bias.

sciencedirect.com/science/articl… (14/23)

Problem 6: Traditional reviews often lack appropriate critical appraisal of included study validity, treating all evidence as equally valid - we know some research is more valid and we need to account for this in the synthesis. (15/23)

Solution 6: Carefully plan and trial a critical appraisal tool before starting the process in full, learning from existing robust critical appraisal tools.
methods.cochrane.org/bias/resources… (16/23)

Problem 7: Inappropriate synthesis (e.g. using vote-counting and inappropriate statistics) can negate all of the preceding systematic effort.

Vote-counting (tallying studies based on their statistical significance) ignores study validity and magnitude of effect sizes. (17/23)

Solution 7: Select the synthesis method carefully based on the data analysed.

Vote-counting should never be used instead of meta-analysis.

Formal methods for narrative synthesis should be used to summarise and describe the evidence base.
citeseerx.ist.psu.edu/viewdoc/downlo… (18/23)

Problem 8: A lack of consistency and error checking (working individually) can introduce errors and biases if a single reviewer makes decisions without consensus.
onlinelibrary.wiley.com/doi/abs/10.100… (19/23)

Solution 8: Have two reviewers screen at least a subset of the evidence base to ensure consistency and shared understanding of the methods before proceeding. Ideally have two reviewers conduct all decisions separately and then consolidate. (20/23)

Summary: There is a lack of awareness and appreciation of the methods needed to ensure systematic reviews are as free from bias and as reliable as possible: demonstrated by recent, flawed, high-profile reviews. (21/23)

We call on review authors to conduct more rigorous reviews, on editors and peer-reviewers to gate-keep more strictly, and the community of methodologists to better support the broader research community. (22/23)

Only by working together can we build and maintain a strong system of rigorous, evidence-informed decision-making in conservation and environmental management. (23/23)