Thread by @AnderRV_

New post! Web Scraping with #Python 101

Learn how to build a web scraper with Python using Requests and BeautifulSoup libraries. We will cover, step-by-step, a scraping process on a job board.

zenrows.com/blog/web-scrap…

#WebScraping might be divided into for main steps:

1. Explore the target site before coding
2. Retrieve the content (HTML)
3. Extract the data you need with selectors
4. Transform and store it for its use

1/ Understand the page you're trying to scrape, not just the content but the structure.

Use DevTools to take a look and inspect its content. What can be scraped, and how is it displayed inside the page.

2/ Obtain the HTML, in our example with Requests. We will focus on static content for simplicity.

The first problems with blocks might appear in this step. The first thing to try: smart rotating proxies such as zenrows.com

3/ Extract the data you are looking for.

Using BeautifulSoup, a library "for pulling data out of HTML," you can select what pieces of information you want to get.

You can use CSS selectors or other solutions like metadata or rich snippets.

4/ Store extracted data in a practical format for your use case, for example, CSV files.

We will use Pandas library to easily convert from an array with results to a file that anyone can open on Excel.

This is just the tip of the iceberg, and there are other things to take into account when building a robust solution:

- Error handling
- Scaling up (more pages, different domains)
- Anti-bot software

Visit the blog post for more details with code examples.

We scrape remotive.io as an example and store some basic info on a CSV file.

zenrows.com/blog/web-scrap…