Thread by @WeAreRLadies

And in my last Twitter thread, I wanted to talk with you about some powerful approaches in #NLP and how we can use both #rstats and #python to unleash them 💪

One possible downside when using the bag of words approach described before is that you often cannot fully take the structure of the language into account (n-grams are one way, but they are often limited).

You also often need many data to successfully train your model - which can be time-consuming and labor intensive. An alternative is to use a pre-trained model. And here comes @Google's famous deep learning model: BERT.

@Google 💡 What is BERT?

BERT is the acronym for bidirectional encoder representation from transformers. To understand how a BERT model works, I like to look at how it understands your text and how you train a BERT model.

@Google Simply speaking, there are three essential components:

✨ First, with BERT, you identify the order of the input.

@Google You give the model information about different embedding layers (the tokens (BERT uses special tokens (for instance, [CLS] and [SEP]) to make sense of the sentence), the positional embedding (where each token is placed in the sentence),

@Google and the segment embedding (which gives you more info about the sentences to which the tokens belong).

@Google And then there is the training:

🏋🏼‍♀️ The first half of the training involves masking world (Mask ML). During the training period, you mask one word at a time and the model learns, which word usually follows.

@Google 🏋🏼‍♀️ During the second half, you train the model to predict the next sentence. This way, the model learns which sentences usually follow each other.

@Google If you want more, there are so many great videos online explaining how a BERT model works ( 📺 bit.ly/bert-deep-lear… - it's all about attention ✨) but what's important to remember: it comes with a pre-trained language model.

@Google You can now use your labeled data to fine-tune the pre-trained model. One way of thinking about it is to think of a student: with the classical bag of words approach, you need to teach the student the language first. 👩🏼‍🏫

@Google With BERT, you have a student who already knows the language but you're teaching the student a specific topic like biology. 🧬

@Google There is a fantastic framework in #python to work with BERT models - 🤗 @huggingface: huggingface.co. It has several pre-trained models available on the website including fantastic tutorials that are very easy to follow.

@Google @huggingface If you're up for using it in #rstats - the great community got you covered. There is a tutorial for everything - and here's one from @rstudio explaining how to use the framework in R with 📦 {reticulate}: blogs.rstudio.com/ai/posts/2020-…

@Google @huggingface @rstudio And another cool thing: Once you understand how BERT works, you can also apply the logic to a variety of text, audio, or video data tasks: huggingface.co/tasks 🥳

@Google @huggingface @rstudio 📝 As the last wrap-up for this week, here's the visual of two possible workflows - comparing bag of words and BERT. They're not so much different as you will see. One approach uses a bag of words and the other approach uses #BERT.

@Google @huggingface @rstudio With both approaches, you load your data but with #BERT you get a specifically pre-trained tokenizer (to make sense of your words) and model as well. You pre-process your text in both cases and then make use of it to perform your task at hand 👩🏼‍💻

@Google @huggingface @rstudio If you want more in-depth insights into how to do it, there's more:

👉 bag of words and {quanteda} 📦: bit.ly/text-mining-qu…

@Google @huggingface @rstudio 👉 BERT with the 🤗 @huggingface framework - particularly the tokenizer pipeline (bit.ly/hf-tokenizer) and the tutorial on fine-tuning (bit.ly/hf-fine-tune) are great to get you started! ✨