Thread by @AnderRV_

New post! Web Scraping with Selenium in Python.

Learn how to navigate and scrape websites using @SeleniumHQ in #Python, even dynamic content, thanks to Javascript Rendering and other available features.

zenrows.com/blog/web-scrap…

After installing and launching the basics, we will start selecting elements using several ways. That will allow us to interact with them to click or fill in forms.

For the particular case of infinite scroll or lazy loaded content, we'll take advantage of sending keyboard inputs.

Another typical use case is to wait for an element to be present. That might happen in dynamic pages loaded via XHR. The initial HTML is just a skeleton with no actual content.

We can wait for the content to be present before starting the data extraction.

Selenium also offers other valuable tricks such as taking screenshots and resizing the window.

That last one plus custom headers is a perfect combo for scraping certain sites. We could, for example, simulate an Android device to get content as a mobile user.

And to avoid blocks, an important part is missing: proxies. Selenium's support is elementary, but libraries (i.e., selenium wire) offer features like auth. Which also allows changing proxy once started.

Anyway, we advocate for smart rotating proxies like zenrows.com.

To increase scraping speed, we can block resources like images or stylesheets. And going a step further, intercept requests and decide programmatically to abort or continue with it.

Last but not least, some actions are easiest to perform on the client-side. And for that, Selenium offers the ability to execute Javascript. `driver.execute_script("return navigator.userAgent")`

As simple as that.

To summarize, Selenium is a great tool to navigate and scrape dynamic content or pages that require Javascript execution.

It's not a tool for all cases but perfect when applied for the right tasks.

But there are still other challenges like crawling at scale and blocks.