Thread by @DataScienceFest

#DSFthegreatindoors presents

Workshop - Text Classification by Transfer learning with Deep Transformers - Depop @depop
A talk by Oduwa Edo-Osagie @odieED, Data Scientist, Depop

#TextMining
#NLP
#AI
#DataScience
#MachineLearning
#DataEngineering
@DataIdols
#DSFthegreatindoors
ND

About the talk

Workshop - Text Classification by Transfer learning with Deep Transformers - Depop @depop
A talk by Oduwa Edo-Osagie @odieED, Data Scientist, Depop

#TextMining
#NLP
#AI
#DataScience
#MachineLearning
#DataEngineering

@DataIdols

#DSFthegreatindoors
ND

What is #TransferLearning?

Text Classification by #TransferLearning with #DeepTransformers @depop
A talk by Oduwa Edo-Osagie @odieED, Snr Data Scientist, @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#MachineLearning
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

1st learn features, then work on the classifier
Text Classification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#MachineLearning
#DataEngineering
@DataIdols
#DSFthegreatindoors
ND

Strategies for transfer learning?
1st learn features, then work on the classifier
Te#xtClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop
#TextMining
#NLP
#AI #CNN
#DataScience
#ML
@DataIdols
#DSFthegreatindoors

#TextClassification by #TransferLearning with #DeepTransformers
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Crowdsourced image labelling that's gamified is a good approach
#TextClassification by #TransferLearning w/ #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop
#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering
@DataIdols
#DSFthegreatindoors
ND

What is a transformer?

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Encoding Transformers

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Batch normalisation helps reduce training time

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Moving onto #BERT

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

BERT Bidirectional Encoder Representations from Transformers
#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering
@DataIdols
#DSFthegreatindoors
ND

BERT has 12 transformer layers & 110 million parameters
Training BERT

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
@DataIdols
#DSFthegreatindoors
ND

Language Modelling Objectives

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Notice word piece embeddings (in image) to save memory

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Now add special tokens: CLS & SEP

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Time for some Q&A before we head to the hands-on session

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Want to know more about BERT?

en.wikipedia.org/wiki/BERT_(lan…

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

The GitHub link for the hands-on
github.com/oduwa/DSF_Depo…

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop
#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Use Python 3.6 or later
#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Create a new virtual environment

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Create an abstraction....

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Line 12 class PaddingInputExample

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Create tokeniser line 50

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Notice line 64 onwards
input_ids input_mask segment_ids
#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Generate tokens circa line 70

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Line 90 note zero-pad

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Next the BERT layer
Subclas 3 layers
#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

The TensorFlow Hub
tensorflow.org/hub

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Line 24 call the BertLayer, notice the kwargs

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Now train the last 10 layers line 33 onwards

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Have you seen

tensorflow.org/hub

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Run pretrain BERTs line 49 onwards
#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Now we build our own BERT
Create a BERT class

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

See Lines 16 & 25

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Now the masks and segements line 25 onwards

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

BERT does feature extraction line 31 onwards

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

We get vectors for CLS i sep
Line 31 Get matrix 768 x 64 x 4
#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Use RELU activation function line 32

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
#DataEngineering

@DataIdols
#DSFthegreatindoors
ND

Now for a visual representation
Use loss=binary_crossentropy L 36
#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop
#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
@DataIdols
#DSFthegreatindoors
ND

Line 41 Use 90% of data for training, 10% for testing

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Line 47 Take text from features

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Line 52 Ready to train the model with 2 epochs

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Line 56 Set batch_size-32, this is another hyper-parameter

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Line 25 We need to initialise the session

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Line 67 initialise BERT with data CSV in git repo (dim 500 x 2)

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Pls RETWEET, SHARE & MENTION 🙏
Pls RETWEET, SHARE & MENTION 🙏

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop
#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML
@DataIdols
#DSFthegreatindoors
ND

FANTASTIC LIVE CODING by @odieED

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Note
val_acc 0.82 (validation accuracy)
Total params 110,302,011

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop
#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#MLS
@DataIdols
#DSFthegreatindoors
ND

We used TF1, easier for beginners
tensorflow.org/versions

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Git repo updated in real-time by Oduwa! Thanks.

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Line 65 Now let's use the model to make spam PREDICTIONS
#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Line 82 has our dummy dataset

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Model is running the predictions

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Notice 0.78 probability the dummy data is spam

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Depop are hiring for a Jnr Data Scientist! Get in touch!

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie @odieED Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

MASSIVE THANKS TO
Oduwa Edo-Osagie @odieED & @depop

Pls RETWEET 🙏🙏

#TextClassification by #TransferLearning with #DeepTransformers @depop
By Oduwa Edo-Osagie Snr Data Scientist @depop

#TextMining
#NLP
#TensorFlow
#AI #CNN
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND