Data analysts should be able to efficiently build datasets using the content on the web.
Here I will show you a use case for a small script I created that allows the user to build corpora of textual information from online blogs.
People working in #seo can definitely use this👇
Here I will show you a use case for a small script I created that allows the user to build corpora of textual information from online blogs.
People working in #seo can definitely use this👇
It is a #Python script that efficiently organizes data in a #pandas dataframe for ease of use and readability.
It leverages Trafilatura, an open source library that is able to read and follow links from the website's sitemap and identify the main content of a page.
It leverages Trafilatura, an open source library that is able to read and follow links from the website's sitemap and identify the main content of a page.
This is particularly important because each website has its own structure and it is often difficult to grab the main content as opposed, for instance, to the sidebar.
Trafilatura uses a series of heuristics to find the main text of the page, and it works pretty damn good.
Trafilatura uses a series of heuristics to find the main text of the page, and it works pretty damn good.
By leveraging pandas, we can dump Trafilatura's output in a spreadsheet style object in Python.
The function below iterates through a list of websites we pass to and builds a clean, usable dataset for our analysis.
Conveniently, columns are just URL and content of the article.
The function below iterates through a list of websites we pass to and builds a clean, usable dataset for our analysis.
Conveniently, columns are just URL and content of the article.
Once you launch the software, you'll also be able to leverage TQDM, a super cool lib in Python that applies progress bars to our applications.
No drawbacks, just pure clarity. Its a no brainer in almost all cases and it also integrates with pandas.
No drawbacks, just pure clarity. Its a no brainer in almost all cases and it also integrates with pandas.
You might be asking...what software are you talking about? Well I wrote a piece on #medium some time ago where I go through the steps of building this software for your own analysis.
You can find it at this link here 👇
medium.com/mlearning-ai/h…
You can find it at this link here 👇
medium.com/mlearning-ai/h…
While experienced Python developers and analysts know well that scraping can be done in many different and efficient ways, this is an introductory script that could help new analysts move their first steps in the field of web #scraping and Python.
It goes without saying that you can and should expand this script as you see fit. See it as a template for starting your own project.
Also, Trafilatura does not work with Javascript websites - in that case the output would be empty. This is how it is coded to work.
Also, Trafilatura does not work with Javascript websites - in that case the output would be empty. This is how it is coded to work.
If you want to scrape content from JS-heavy websites, I suggest you use #playwright.
It is a Selenium on steroids - it is fast to setup and a breeze to get it going. Try it out.
Now go out there and build your own dataset!
It is a Selenium on steroids - it is fast to setup and a breeze to get it going. Try it out.
Now go out there and build your own dataset!
If you want to reach to know more, please do. I reply to DMs here and on LinkedIn.
If you want to follow my content creation, sign up on my medium newsletter.
If you are Italian, visit diariodiunanalista.it and sign up there too
If you want to follow my content creation, sign up on my medium newsletter.
If you are Italian, visit diariodiunanalista.it and sign up there too
If you want to share your project with me and want my opinion on it, drop me a message. I'll be happy to follow up with you.
Stay tuned 💪
Stay tuned 💪
• • •
Missing some Tweet in this thread? You can try to
force a refresh
