, 15 tweets, 5 min read Read on Twitter
Here's a thread on what's very wrong about using only Google Scholar for a literature view, as done by Buribalova et al. in this recent review: onlinelibrary.wiley.com/doi/full/10.11… #SystematicReview #EvidenceSynthesis
Problem 1: Their search string is flawed. The authors say they used a 'systematic' search for articles on Google Scholar. However, their search will not work as intended because:
a) GS doesn't support Boolean operators (AND, OR, NOT); b) you cannot nest more than one substring (bracketed set of synonyms) in a search; c) they have not nested geographical synonyms within brackets...
... d) they have missed inverted commas around phrases; e) GS doesn't support asterisks (wildcards).
Problem 2: The authors' search terms are FAR from comprehensive. Any research lacking the terms (impact OR effect) or (tropical OR Asia OR Africa OR South America) - researchers could easily describe their work in other terms, for example referring to the country instead.
Problem 3: The authors use only one source of evidence. They have therefore certainly missed evidence, since GS has been shown to be insufficient as a standalone resource in reviews: journals.uic.edu/ojs/index.php/…
Problem 4: They screened only the first 1000 results of each search. In doing so, they are relying solely on Google's relevance algorithm to rank results - anything deemed to be of low relevance is not displayed in the top 1000. The authors recognise this bias but forge ahead.
Problem 5: The authors fundamentally misunderstand the risks of presenting data in a way that allows easy vote-counting - indeed, their visualisations make it impossible not to vote-count! This is wrong because it ignores study quality and the magnitude of the effect.
Their visualisations are pretty, but based on poor science and promote vote-counting, a method known to be unreliable.
Problem 6: The authors included only peer-reviewed research and a select few technical reports, but provide no details about how these were judged to be of low bias, making this aspect of their methods unrepeatable.
Problem 7: There is no discussion of testing the inclusion criteria or performing consistency checking between reviewers, so we cannot be sure the inclusion decisions were appropriate or repeatable.
Problem 8: The authors separate meta-analyses from systematic reviews, but MA is a quantitative synthesis method, not a type of review (again demonstrating their misunderstanding of terminology and methodology). [This problem is perhaps controversial]
Problem 9: The authors justify their approach by stating that the lack of repeatability and comprehensiveness afforded by Google Scholar is only one of a suite of potential biases, so it's really not that big of a deal. What the hell, why not just cherry pick then?
There are certainly MANY more problems with their approach, but I've now lost the will to live... If you want to see good examples of environmental evidence synthesis, check out evidencesynthesis.org and @EnvEvidence.
Oops - that should have been environmentalevidence.org - I erred in my rage...
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Neal Haddaway
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!