In this paper in @NatureEcoEvo we outline 8 major problems that can occur with traditional ways of reviewing the literature, and provide concrete advice on how to avoid them:

This thread outlines our key points! (1/23)
#EvidenceSynthesis #SystematicReview
Traditional ways of reviewing the literature may be susceptible to bias and end up giving us incorrect conclusions.

This is of particular concern when reviews address key policy- and practice- relevant questions. (2/23)
Systematic reviews aim to maximise rigour and minimise susceptibility to bias using rigorous methods.

But despite these methods being available, poor reviews are still published: these could be improved with a few key processes; some not prohibitively costly. (3/23)
Here follow 8 problems with traditional literature reviews, each with suggested solutions. (4/23)
Problem 1: Lack of relevance - limited stakeholder engagement can produce a review that is of limited practical use to decision-makers (5/23)
Solution 1: Stakeholders can be identified, mapped and contacted for feedback and inclusion without the need for extensive budgets - check out best-practice guidance: (6/23)
Problem 2: Mission creep - reviews that don't publish their methods in an a priori protocol can suffer from shifting goals and inclusion criteria. (7/23)
Solution 2: carefully design and publish an a priori protocol that outlines planned methods for searching, screening, data extraction, critical appraisal and synthesis in detail. Make use of existing organisations to support you (e.g. @EnvEvidence;…) (8/23)
Problem 3: A lack of transparency/replicability in the review methods may mean that the review cannot be replicated (a central tenet of the scientific method!)… (9/23)
Solution 3: Be explicit, and make use of high-quality guidance and standards for review conduct (e.g. CEE Guidance,…) and reporting (PRISMA, or ROSES, (10/23)
Problem 4: Selection bias (where included studies are not representative of the evidence base) and a lack of comprehensiveness (an inappropriate search method) can mean that reviews end up with the wrong evidence for the question at hand.… (11/23)
Solution 4: Carefully design a search strategy with an info specialist; trial the search strategy (against a benchmark list); use multiple bibliographic databases/languages/sources of grey literature; publish search methods in an a priori protocol for peer-review. (12/23)
Problem 5: The exclusion of grey literature and failure to test for evidence of publication bias can result in incorrect or misleading conclusions.… (13/23)
Solution 5: Include attempts to find grey literature, including both 'file-drawer' (unpublished academic) research and organisational reports.

Test for possible evidence of publication bias.… (14/23)
Problem 6: Traditional reviews often lack appropriate critical appraisal of included study validity, treating all evidence as equally valid - we know some research is more valid and we need to account for this in the synthesis. (15/23)
Solution 6: Carefully plan and trial a critical appraisal tool before starting the process in full, learning from existing robust critical appraisal tools.… (16/23)
Problem 7: Inappropriate synthesis (e.g. using vote-counting and inappropriate statistics) can negate all of the preceding systematic effort.

Vote-counting (tallying studies based on their statistical significance) ignores study validity and magnitude of effect sizes. (17/23)
Solution 7: Select the synthesis method carefully based on the data analysed.

Vote-counting should never be used instead of meta-analysis.

Formal methods for narrative synthesis should be used to summarise and describe the evidence base.… (18/23)
Problem 8: A lack of consistency and error checking (working individually) can introduce errors and biases if a single reviewer makes decisions without consensus.… (19/23)
Solution 8: Have two reviewers screen at least a subset of the evidence base to ensure consistency and shared understanding of the methods before proceeding. Ideally have two reviewers conduct all decisions separately and then consolidate. (20/23)
Summary: There is a lack of awareness and appreciation of the methods needed to ensure systematic reviews are as free from bias and as reliable as possible: demonstrated by recent, flawed, high-profile reviews. (21/23)
We call on review authors to conduct more rigorous reviews, on editors and peer-reviewers to gate-keep more strictly, and the community of methodologists to better support the broader research community. (22/23)
Only by working together can we build and maintain a strong system of rigorous, evidence-informed decision-making in conservation and environmental management. (23/23)

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Neal Haddaway

Neal Haddaway Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @nealhaddaway

9 Oct
Check out our new paper on the need for better understanding of academic searching:…

#InformationRetrieval #MedLibs #EvidenceSynthesis

A thread with our key points... (1/20) Image
We agree with others that we now face an 'information crisis' (#infodemic). There is SO much published research we need to find and digest.

Doing this reliably requires systematic review approaches, but even then, it's hugely challenging to find all relevant research.
(2/20) Image
Academic searching/information retrieval is an art form, and there is no 'perfect' search strategy - it takes careful planning and requires substantial skill and training.

These searches are highly complex and must be used in fit-for-purpose bibliographic databases. (3/20) Image
Read 20 tweets
8 Oct
I've built an R package to scrape search results from up to 100 pages of hits from Google Scholar:…

It's probably a bit buggy, but feedback greatly appreciated!

Here's a thread on how it works...

#AcademicTwitter #InformationRetrieval #MedLibs
1) It generates a set of up to 100 URLs, each corresponding to a page of visible search results Image
2) It then downloads each of these pages of results as an HTML file Image
Read 6 tweets
24 Apr 19
Here's a thread on what's very wrong about using only Google Scholar for a literature view, as done by Buribalova et al. in this recent review:… #SystematicReview #EvidenceSynthesis
Problem 1: Their search string is flawed. The authors say they used a 'systematic' search for articles on Google Scholar. However, their search will not work as intended because:
a) GS doesn't support Boolean operators (AND, OR, NOT); b) you cannot nest more than one substring (bracketed set of synonyms) in a search; c) they have not nested geographical synonyms within brackets...
Read 15 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!