Thread by @WeAreRLadies

The curation week is almost over and I would like to thank everyone for joining the discussions this week! It’s been a blast 🥳
If you enjoyed this week, feel free to reach out on Twitter (@cosima_meyer) or GitHub (github.com/cosimameyer/) ✨

@cosima_meyer I feel very honored that I had the chance to talk with you about the things I enjoy doing and I cannot wait to learn more from the upcoming curators - the lineup looks amazing! 💜

@cosima_meyer If you missed a Twitter thread this week, head over to @pilizalde's amazing thread where she collected all of them (I love the GitHub emoji 😺)

👇

https://twitter.com/pilizalde/status/1570741209806798854

@cosima_meyer @pilizalde But before I leave and head over to do some serious vinyl shopping 🎶, I want to talk about #NLP with you 💬

@cosima_meyer @pilizalde 💡 What is NLP?
NLP is short for Natural Language Processing and it helps make sense of a difficult data type: written text.

@cosima_meyer @pilizalde 📑 But let's first start with the basic concepts and add a Gilmore Girls flavor to it - because who would be a better use case than those who have a seemingly endless vocabulary?

@cosima_meyer @pilizalde Here are all the terms that you need to know to get started 👇

@cosima_meyer @pilizalde ✨ Corpus: When you have your text data ready, you have your corpus. It's a collection of documents.

@cosima_meyer @pilizalde ✨ Tokens: Define each word in a text (but it could also be a sentence, paragraph, or character).

@cosima_meyer @pilizalde ✨ Tokenization: When you hear the word tokenization, it means that you are splitting up the sentences into single words (tokens) and turning them into a bag of words. You can take this quite literally - a bag of words does not really take the order of the words into account.

@cosima_meyer @pilizalde There are ways to account for the order using n-grams (so for instance a bigram would leave the sentence "Rory lives in a world of books" as "Rory lives", "lives in", "in a", "a world", "world of", "of books") but it's limited.

@cosima_meyer @pilizalde ✨ Document-feature matrix (DFM): To generate the DFM you first split the text into its single terms (tokens), then count how frequently each token occurs in each document.

@cosima_meyer @pilizalde ✨ Stemming: With stemming, you are getting the stem of the word.

@cosima_meyer @pilizalde ✨ Lemmatization: With lemmatization, it's slightly different. Instead of "stud" (which would probably be the stem of the study terms), you end up with a meaningful stem - "study" 🥳

@cosima_meyer @pilizalde The overview also describes a typical workflow with the bags-of-word approach nicely.

👉 You typically load the #data,

👉 tokenize it (and turn it into a bag full of words),

👉 pre-process it by stemming it (and removing stop words and a bit more)

@cosima_meyer @pilizalde 👉 count the single words to turn the count into a DFM (document-feature matrix) - and now you're ready to go! 🎉

@cosima_meyer @pilizalde From here on, you can do multiple tasks - for instance, you can perform #supervised tasks with dictionary approaches or classify the sentiment or topics. But you can also use it to perform #unsupervised tasks like structural topic models. The possibilities are almost endless.

@cosima_meyer @pilizalde If you're up for more on how to use 📦 {quanteda} in #rstats on these tasks, here is more from a hands-on workshop that I had the honor to give @RLadiesBergen

I deployed the code in a readable and downloadable #Rmd file for you to use 👇

@cosima_meyer @pilizalde @RLadiesBergen It contains everything from detailed terms and concepts description over data preparation to using supervised and unsupervised approaches.

💻 The deployed code can be accessed here: nlp-bergen.netlify.app

And if you want me to talk you through the code, here's a recording and the slides to flip through 🤓

📺 @RLadiesBergen:

📑 Slides: cosimameyer.com/slides/nlp-rla…

@RLadiesBergen If you're up for more verbose, it's also based on a blog post by @cbpuschmann and myself:
📑 @mzes_ssdl: bit.ly/text-mining-qu…