Thread by @GiordMarco96

Scraping is super important for a lot of tasks.

#SEO Specialists usually crawl their websites but scraping is not always one of the tools in their arsenal.

A short thread on what you can do with scraping and what's the difference with crawling. 🧵

In plain English, crawling is the act of discovering URLs on a website, so following the links on a page is extremely important.

Scraping is the extraction of data from a website, not all of them of course, just those of interest to achieve a goal.

As we know, crawling is aimed at creating indices of pages.

We care about scraping because we may be interested in extracting specific information from a page.

Now, let's move on to some SEO use cases:

- Check price updates
- Get sitemaps/RSS feeds for content strategy
- Competitor monitoring
- Data analysis and getting data in general

I check sitemaps or feeds to understand the publishing rate of my competitors.

You can also get the latest news and much more, actually.

holisticseo.digital/python-seo/con…

No need to guess content velocity, just check it.

If they don't have a sitemap, things get way more complex.

And here you can just scrape by folder or find a way to create your custom scraper.

You may also want to scrape pages to apply NLP after.

Example: Scrape a page and then analyze the entities.

Very simple and effective, I will post an example in the next threads.

I know what you're thinking, scraping can be used for less ethical purposes.

Many still view it with negative meaning, and I don't blame them either.

Anyway, it's an important task that can open you many opportunities!

And yes, this can be used to get data that you'll use for Programmatic SEO.

I am not an expert on that SEO branch but ofc it works!

Gathering data is one of the use cases I mentioned before.

Follow me for threads, tips and case studies (coming soon) about SEO, content and #Python/data.

If you liked this thread, consider liking and retweeting it!