Thread by @nealhaddaway

Here's a thread on what's very wrong about using only Google Scholar for a literature view, as done by Buribalova et al. in this recent review: onlinelibrary.wiley.com/doi/full/10.11… #SystematicReview #EvidenceSynthesis

Problem 1: Their search string is flawed. The authors say they used a 'systematic' search for articles on Google Scholar. However, their search will not work as intended because:

a) GS doesn't support Boolean operators (AND, OR, NOT); b) you cannot nest more than one substring (bracketed set of synonyms) in a search; c) they have not nested geographical synonyms within brackets...

... d) they have missed inverted commas around phrases; e) GS doesn't support asterisks (wildcards).

Problem 2: The authors' search terms are FAR from comprehensive. Any research lacking the terms (impact OR effect) or (tropical OR Asia OR Africa OR South America) - researchers could easily describe their work in other terms, for example referring to the country instead.

Problem 3: The authors use only one source of evidence. They have therefore certainly missed evidence, since GS has been shown to be insufficient as a standalone resource in reviews: journals.uic.edu/ojs/index.php/…

Problem 4: They screened only the first 1000 results of each search. In doing so, they are relying solely on Google's relevance algorithm to rank results - anything deemed to be of low relevance is not displayed in the top 1000. The authors recognise this bias but forge ahead.

Problem 5: The authors fundamentally misunderstand the risks of presenting data in a way that allows easy vote-counting - indeed, their visualisations make it impossible not to vote-count! This is wrong because it ignores study quality and the magnitude of the effect.

Their visualisations are pretty, but based on poor science and promote vote-counting, a method known to be unreliable.

Problem 6: The authors included only peer-reviewed research and a select few technical reports, but provide no details about how these were judged to be of low bias, making this aspect of their methods unrepeatable.

Problem 7: There is no discussion of testing the inclusion criteria or performing consistency checking between reviewers, so we cannot be sure the inclusion decisions were appropriate or repeatable.

Problem 8: The authors separate meta-analyses from systematic reviews, but MA is a quantitative synthesis method, not a type of review (again demonstrating their misunderstanding of terminology and methodology). [This problem is perhaps controversial]

Problem 9: The authors justify their approach by stating that the lack of repeatability and comprehensiveness afforded by Google Scholar is only one of a suite of potential biases, so it's really not that big of a deal. What the hell, why not just cherry pick then?

There are certainly MANY more problems with their approach, but I've now lost the will to live... If you want to see good examples of environmental evidence synthesis, check out evidencesynthesis.org and @EnvEvidence.

Oops - that should have been environmentalevidence.org - I erred in my rage...