Thread by @AnderRV_

New post! Stealth Web Scraping in #Python: Avoid Blocking Like a Ninja. We share the best techniques for massive scale scraping.
From the basic, such as avoiding rate limits or adding proxies, to more complex as full set of headers or behavioral patterns.
zenrows.com/blog/stealth-w…

For the basic defensive protections, rotating proxies with the correct headers should be enough.

For a bit more complex ones, maybe residential IPs are necessary.

Captchas can be solved nowadays, but it is best to bypass them. The same applies to login or paywalls.

Sending real-world User-Agents is important. But not enough, since there are other headers involved, i.e. sec-ch-ua or sec-fetch-dest.

They all should be used together, to avoid suspicion.

When using Headless Browsers, headers are important but be careful with Javascript check. If your header does not match `navigator.userAgent`, you can be tagged as a potential bot. Or directly banned, depending on their criteria.

For state-of-the-art solutions, you should take into account even behavioral patterns. It means that real users do not browse through the entire page in less than a second. Or fill a form in an instant.

Moving the mouse, scroll at a human pace, clicking links, and several other actions can confuse the defensive systems.

But repetitive behaviour can be also tracked. Visiting the same pages in order and exactly the same actions, can also be flagged.

Shuffle some URLs, change the order, modify scrolling patters, or add some delays.

These actions are not enough on their own, but a combination of them can be next to impossible to detect.
Sometimes implementing all of these is an overkill, it all depends on the other side.
And above all, be a good internet citizen. Do not use this to perform malicious actions.