This week, let's talk about model building a little.
In the TensorFlow world, the simplest way of building a model is with a Sequential model.
But what is it and how to do it?
[4 minutes]
1/9🧵
First, let's go over some basics.
The goal here is to build a Neural Network, or in other words, create a set of neurons, distributed in layers and connected by weights.
Each Layer applies some computation on the values or tensors it receives.
2/9🧵
The simplest way to create this NN is to just do a plain stack of layers where each layer has exactly one input and one output.
This is exactly what a Sequential model does
3/9🧵
After you have your model, you can access each layer and it's weights.
You might want to do that if you want to do feature extraction for example
4/9🧵
You can also visualize the model structure using the summary method
This can you give some insights of the intermediate shapes and help fix issues
💡:layer1 and layer2 (or any layer that is not input or output) is also known as a hidden layer
5/9🧵
If you want a more visual version, you can also use tf.keras.utils.plot_model(model)
This shows a more visual version of the model.
💡: Both this and the summary will work even if you build your model not using a Sequential model.
6/9🧵
You should NOT use a Sequential model when:
• Your model or any layer needs multiple inputs or outputs
• You need a non-linear model
• You need layer sharing
Sometimes you need to build a Machine Learning model that cannot be expressed with the Sequential API
For these moments, when you need a more complex model, with multiple inputs and outputs or with residual connections, that's when you need the Functional API!
[2.46 min]
1/8🧵
The Functional API is more flexible than the Sequential API.
The easiest way to understand is to visualize the same model created using the Sequential and Functional API
2/8🧵
You can think of the Functional API as a way to create a Directed Acyclic Graph (DAG) of layers while the Sequential API can only create a stack of layers.
Functional is also known as Symbolic or Declarative API
One very interesting task on the NLP fields is text generation.
There are very advanced techniques and a lot of research on it and even business based solely on it!
But how does it work?
[7.47min]
[I guarantee it's a much better read then doom scrolling!!!]
1/11🧵
Let's think: what a model would have to do to generate text?
The rationale is, as humans we form sentences by trying to create a sequence of words that makes sense.
The less random this sequence looks like, the better the output text is and closer to human like.
2/11🧵
Here is where ML can help.
A model should learn how to combine the words the best way possible.
The simplest way to teach this is: given a sentence, hide the last word and let the model try to guess it.
The loss function measures how good the model's guess is.
Sometimes you need to create your own model for your specific data corpus (eg: legal, science, medical texts)
To create your own model, AutoML Natural Language can help you:
2/4🧵
If you want to build everything from scratch, then you'll need:
• a language embedding (like BERT, ELMO, USE) and #TFHub have all you need
• a dataset and this github.com/juand-r/entity… can help you find one
Encoding text in numbers is a very important part of NLP as the better this can be done, the better are the possible results!
Word embedding works but they don't have the full context of the sentence.
This is where BERT comes in
But what is BERT?
1/9🧵
When we do word embedding, both sentences
• They are running a test
• They are running a company
Will have very similar embeddings but the meaning of both sentences are very different. Without this context, the model using this encoding will be blind to the context
2/9🧵
This is where Bidirectional Encoder Representations from Transformers (BERT) comes in play!
It is a Transformer-based network created in 2018 and
takes into account the context of the word's occurrence. For the previous example, it gives very different embeddings.