New post! DOs and DON'Ts of Web #Scraping
Learn how to create better web scrapers by following best practices and avoiding common mistakes. Choose the right approach for the job thanks to these tips.
zenrows.com/blog/dos-and-d…
Learn how to create better web scrapers by following best practices and avoiding common mistakes. Choose the right approach for the job thanks to these tips.
zenrows.com/blog/dos-and-d…
1. DO Rotate IPs
The most common anti-scraping solution is to ban by IP. By using proxies, you'll avoid showing the same IP in every request and thus increasing your chances of success. You can code your own or use a Rotating Proxy like zenrows.com.
The most common anti-scraping solution is to ban by IP. By using proxies, you'll avoid showing the same IP in every request and thus increasing your chances of success. You can code your own or use a Rotating Proxy like zenrows.com.
2. DO Use Custom User-Agent
Overwrite your client's User-Agent, or you'll risk sending something like "curl/7.74.0".
But always sending the same UA might be suspicious, too, so you need a vast and updated list.
Overwrite your client's User-Agent, or you'll risk sending something like "curl/7.74.0".
But always sending the same UA might be suspicious, too, so you need a vast and updated list.
3. DO Research Target Content
Look at the source before coding. Many sites expose data through metadata, attributes, or hidden inputs. There are options beyond CSS selectors and are usually more maintainable. Search also for XHR requests.
Look at the source before coding. Many sites expose data through metadata, attributes, or hidden inputs. There are options beyond CSS selectors and are usually more maintainable. Search also for XHR requests.
4. DO Parallelize Requests
To scale up, sequential requests will not be enough.
The next step is to get several URLs simultaneously and have a queue that allows adding items to be processed later.
To scale up, sequential requests will not be enough.
The next step is to get several URLs simultaneously and have a queue that allows adding items to be processed later.
5. DON'T Use Headless Browsers for Everything
They are great but not a silver bullet. They bring a resource overhead and slow down the scraping process.
Test first plain HTML and only go to Headless Browsers if necessary.
They are great but not a silver bullet. They bring a resource overhead and slow down the scraping process.
Test first plain HTML and only go to Headless Browsers if necessary.
6. DON'T Couple Code to Target
Apply Software Engineering good practices.
Separate general crawling code (get HTML, parse it, enqueue links) from the target-specific (selectors, scraped data, URL structure).
Apply Software Engineering good practices.
Separate general crawling code (get HTML, parse it, enqueue links) from the target-specific (selectors, scraped data, URL structure).
7. DON'T Take Down your Target Site
Be mindful of the scale of your scraping and the size of your targets. It's not the same as Amazon or a small store.
Mainly two rules: do not scrape disallowed pages and obey Crawl-Delay.
Be mindful of the scale of your scraping and the size of your targets. It's not the same as Amazon or a small store.
Mainly two rules: do not scrape disallowed pages and obey Crawl-Delay.
8. DON'T Mix Headers from Different Browsers
Browsers send several headers with a set format that varies from version to version. And modern anti-bot solutions check those.
The only solution is to have huge lists with full-set headers, not just UA.
Browsers send several headers with a set format that varies from version to version. And modern anti-bot solutions check those.
The only solution is to have huge lists with full-set headers, not just UA.
Rotating IPs and good headers will allow you to scrape most websites. Use headless browsers only when necessary and apply Software Engineering good practices.
You probably don't need 100% accuracy - settle for the level that works for you. Overengineering might turn against you.
You probably don't need 100% accuracy - settle for the level that works for you. Overengineering might turn against you.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
