Git scraping: track changes over time by scraping to a Git repository
simonwillison.net/2020/Oct/9/git…
simonwillison.net/2020/Oct/9/git…
I realized I've been using the term "git scraping" for a while without publishing a clear explanation of what I mean by the term
Git scraping = writing scrapers that record their results in a Git repository in order to track changes to the scraped resource over time
Git scraping = writing scrapers that record their results in a Git repository in order to track changes to the scraped resource over time
The Hacker News thread has some great examples of people who've discovered the same pattern news.ycombinator.com/item?id=247329…
I particularly like this one - it's been running for three years scraping bug bounty target domains from sites like @Hacker0x01 github.com/arkadiyt/bount…
I particularly like this one - it's been running for three years scraping bug bounty target domains from sites like @Hacker0x01 github.com/arkadiyt/bount…
I just noticed you can sort the git-scrapers topic on GitHub by "recently updated", which gives a live view of the scrapers people have built - there's a few new ones on there since I published my article this morning! github.com/topics/git-scr…
Handy tip from Hacker News: git show --color-words gives a more detailed word-level diff. Here's that run against the latest diff recorded by github.com/simonw/ca-fire… 
• • •
Missing some Tweet in this thread? You can try to
force a refresh
