Tweet

Neal R Haddaway

Apr 17 • 8 tweets • 4 min read

🚨greylitsearcher update🚨

Significant upgrade to greylitsearcher! Now supports advanced searching of Google AND provides a search report text file.

estech.shinyapps.io/greylitsearche…

Here's some details... 🧵

#medlibs #InformationRetrieval @SystematicSearching #EvidenceSynthesis

You can now build complex Google site: searches the same way you can in Google Scholar...

And you can search across multiple sites simultaneously... (just be sure they support Google site: searching)

developers.google.com/search/docs/ad…

greylitsearcher builds a set of links for each page of search results for each site you enter:

It then grabs the HTML for each page of results, pausing to remain 'friendly' to the bots at Google HQ:

The grabbed HTMLs are then scraped for patterned data and the results are tabulated for each source page, showing titles, descriptions and links:

As well as downloading the CSV file of search results, you can save a search history record, showing exactly what you searched for and when, and which URLs were grabbed and scraped.

If you have any feedback or troubles using greylitsearcher, please let me know by adding an issue on the GitHub repository here:
github.com/nealhaddaway/g…

• • •

Missing some Tweet in this thread? You can try to force a refresh

This Thread may be Removed Anytime!

Twitter may remove this content at anytime! Save it as PDF for later use!

More from @nealhaddaway

Neal R Haddaway

@nealhaddaway

May 5

Are you attending the #Stockholm50 academic conference on June 1st? Look out for my photography exhibition, 'Hope? and how to grieve for the planet'.

#Photography #DocumentaryPhotography #PhotoStory #ClimateCrisis #ClimateChange #ClimateEmotions

nealhaddaway.com/hope

(1/4)

My project asks how environmental scientists and communicators deal with the emotional impact of working with alongside #EnvironmentalCrises.
nealhaddaway.com/hope

Many of us a grieving for our planet, and sharing our #emotions could be vital in the continuing battle.

(2/4)

I'm really honoured to have my work selected alongside several other arts projects aiming to complement the academic conference.

If you're in Stockholm (01/06/22) and want to come, send me a DM for a (free) registration link.

#Stockholm50

(3/4)

Read 4 tweets

Neal R Haddaway

@nealhaddaway

Oct 12, 2020

@NatureEcoEvo

In this paper in @NatureEcoEvo we outline 8 major problems that can occur with traditional ways of reviewing the literature, and provide concrete advice on how to avoid them: rdcu.be/b8pp0

This thread outlines our key points! (1/23)
#EvidenceSynthesis #SystematicReview

Traditional ways of reviewing the literature may be susceptible to bias and end up giving us incorrect conclusions.

This is of particular concern when reviews address key policy- and practice- relevant questions. (2/23)

Systematic reviews aim to maximise rigour and minimise susceptibility to bias using rigorous methods.

But despite these methods being available, poor reviews are still published: these could be improved with a few key processes; some not prohibitively costly. (3/23)

Read 23 tweets

Neal R Haddaway

@nealhaddaway

Oct 9, 2020

Check out our new paper on the need for better understanding of academic searching:
onlinelibrary.wiley.com/doi/10.1002/jr…

#InformationRetrieval #MedLibs #EvidenceSynthesis

A thread with our key points... (1/20)

We agree with others that we now face an 'information crisis' (#infodemic). There is SO much published research we need to find and digest.

Doing this reliably requires systematic review approaches, but even then, it's hugely challenging to find all relevant research.
(2/20)

Academic searching/information retrieval is an art form, and there is no 'perfect' search strategy - it takes careful planning and requires substantial skill and training.

These searches are highly complex and must be used in fit-for-purpose bibliographic databases. (3/20)

Read 20 tweets

Neal R Haddaway

@nealhaddaway

Oct 8, 2020

I've built an R package to scrape search results from up to 100 pages of hits from Google Scholar: github.com/nealhaddaway/G…

It's probably a bit buggy, but feedback greatly appreciated!

Here's a thread on how it works...

#AcademicTwitter #InformationRetrieval #MedLibs

1) It generates a set of up to 100 URLs, each corresponding to a page of visible search results

2) It then downloads each of these pages of results as an HTML file

Read 6 tweets

Neal R Haddaway

@nealhaddaway

Apr 24, 2019

Here's a thread on what's very wrong about using only Google Scholar for a literature view, as done by Buribalova et al. in this recent review: onlinelibrary.wiley.com/doi/full/10.11… #SystematicReview #EvidenceSynthesis

Problem 1: Their search string is flawed. The authors say they used a 'systematic' search for articles on Google Scholar. However, their search will not work as intended because:

a) GS doesn't support Boolean operators (AND, OR, NOT); b) you cannot nest more than one substring (bracketed set of synonyms) in a search; c) they have not nested geographical synonyms within brackets...

Read 15 tweets

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Share this page!

Neal R Haddaway

People who liked this thread also liked...

Try unrolling a thread yourself!

More from @nealhaddaway

Neal R Haddaway

Neal R Haddaway

Neal R Haddaway

Neal R Haddaway

Neal R Haddaway

Did Thread Reader help you today?

Don't want to be a Premium member but still want to support us?