Thread by @gusthema

One very interesting task on the NLP fields is text generation.

There are very advanced techniques and a lot of research on it and even business based solely on it!

But how does it work?

[7.47min]
[I guarantee it's a much better read then doom scrolling!!!]

1/11🧵

Let's think: what a model would have to do to generate text?

The rationale is, as humans we form sentences by trying to create a sequence of words that makes sense.

The less random this sequence looks like, the better the output text is and closer to human like.

2/11🧵

Here is where ML can help.

A model should learn how to combine the words the best way possible.
The simplest way to teach this is: given a sentence, hide the last word and let the model try to guess it.

The loss function measures how good the model's guess is.

3/11🧵

This is what's called creating n-grams
You can do by word, by letter, phonemes, etc

en.wikipedia.org/wiki/N-gram

4/11🧵

To create your own dataset is just lot's of text that we know is written with proper grammar for our language. For example all Shakespeare writing

How about the model architecture?

5/11🧵

We need, as mentioned before, a text embedding.

We also need layers that understand sequences like Recurrent Neural Networks (RNN) or even better, LSTM (Long Short-Term Memory)

Here is a full example: tensorflow.org/tutorials/text…

6/11🧵

The Verse by Verse experiment shows how far this idea can go: sites.research.google/versebyverse/

The presented solution is much more complex but now you understand the basics!

7/11🧵

Text Generation is almost synonyms to OpenAI's GPT-3

I guess by now you've seen the amazing text that can be generated by just giving a beginning for the model to follow up

Code: github.com/openai/gpt-3
Paper: arxiv.org/abs/2005.14165

8/11🧵

Generating text is more than a technical challenge.
The results of a model can be used for good and there are already startups that are based on GPT-3!

theverge.com/2021/3/29/2235…

But it can also be used for not so noble ends.

9/11🧵

One interesting consequence of huge models that generate text is: These models are trained using the internet as source. The more these models generate text and this is published on sites, the more the model will use itself as training material later!
👀🤔

10/11🧵

Sorry for the long thread but this subject is fascinating for me!

And no, this thread wasn't automatically generated by ML…. or was it?

🤖🤔👀😱

11/11🧵