Transfer learning: Pre-training and Fine tuning
Arun Prakash A
Movie Sentiment Classification Task
LSTM
Model
A stunning story with
a
lots
of
twists and..
\begin{bmatrix}
0 \\
0\\
1\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
1 \\
0\\
0\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
0 \\
0\\
0\\
0\\
1\\
\end{bmatrix}
\hat{y}=+
Supervised learning: Train the entire model and learn the word representation by randomly initializing the weights
It is a legitimate attempt. However, what could be drawbacks of this approach?
Word Representation
Learnable Components
Product Sentiment Classification Task
LSTM
Model
The
trimmer
is
noisy
, not
customiz-able
\begin{bmatrix}
0 \\
0\\
1\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
1 \\
0\\
0\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
0 \\
0\\
0\\
0\\
1\\
\end{bmatrix}
\hat{y}=-
The knowldge gained in the previou task is not transferable! Though the underlying task remains same
We have to train the model from scratch again (that is, randomly initialize weights) using new dataset.
Word Representation
Learnable Components
Moreover, the vocabulary may not be the same (say, product name, customization, ..)
Is transfer learning possible?
That is, can we (somehow) use the parameters from a model trained on a smilar task?
Fine tune those paramters on the new task
Motivation from vision
Suppose we train a CNN model to recognize a cat
98 \%
2 \%
cat
not \ cat
ConvNet
We can collect thousands of samples for cat images and thousands for not-cat images and train the model.
Suppose we change the task to object detection. That is, detect the boundary of cat in an image
In other words, can we transfer the learning?
This is a related problem, only difference is that the label is now the coordinates of the bounding box
Can we make use of the ConvNet trained to classify the cat image?
(x_1,y_1)
(x_2,y_2)
Feature Extractor
Discriminator
Transfer learning
98 \%
2 \%
cat
not \ cat
Feature Extractor
Regressor
Transfer learning
Fine tune the parameters of (Multiple) conv layers
random initialize the
parameters for new layers
(x_1,y_1)
(x_2,y_2)
x_1
x_2
y_1
y_2
CNN Trained on
ImageNet
(Millions of Samples)
Finetune for other downstream
Vision tasks
(thousands of samples)
Transfer Learning
(as weight initializer)
(Supervised) Pre-training
Fine Tuning
Outputs from Conv layers act as feature representations.
This can be further fine-tuned for similar tasks like classification, object detection..
These feature representations are learned in a supervised setting
Does it work for text data?
One solution we can think of is using word2vec embedding and fine-tune it as follows
Fine Tune
Train from scratch
Learnable Components
A stunning story with
a
lots
of
twists and..
\begin{bmatrix}
0 \\
0\\
1\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
1 \\
0\\
0\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
0 \\
0\\
0\\
0\\
1\\
\end{bmatrix}
LSTM
Model
\hat{y}=+
Word2Vec Embedding
Yes. It (almost) solves the problem of change in vocabulary in down stream tasks.
How does self-supervised pre-training help?
It makes use of a large scale unlabelled data.
Getting labelled data is costly and time consuming
-
100 billion words
-
3 million words in the vocabulary
Limitations:
The context of a word in a sentence is not captured in the representation
The transfer learning happens only in the embedding (first) layer.
A common thread
Image of \(Bert\)
256
256
An Image is composed of pixels at an elementary level.
An Image is composed of features at a higher level
A sentence is composed of characters at an elementary level.
A sentence is composed of words at a higher level
The feature representations learned by CNN can be transferred and fine-tuned for other tasks
The word representation learned by word2vec can also be transferred and fine-tuned
However, there is some important differences.
he sat on the bank of the river and watched the currents
he went to the bank to check balance in his current account
30 nearest nighbours to the word bank in word2vec embedding space is show in the figure
Let's consider example sentences to illustrate the need for contexual representation
The vector representation for the word bank doesn't change based on the context
Therefore, the model should take the responsibility to understand the words in its context.
Leraning the context in the supervised setting is once again a limitation.
Limitations of Word2Vec
\cdots
The mouse is too bad and the response is very poor
Task Specific Model
Word2Vec
\cdots
\cdots
(Computer)mouse
(lab)mouse
keyborad
cat
word2vec can't have two representations for a word.However, a word can have multiple meanings based on its sentence-context.
Recall, word2vec learns representation using context of a word w.r.t its surrounding words within a fixed size window (independent of order of words)
A mouse chases a cat
Moreover, it doesn't include the positional information. Think of question-answering system where the position of words is important
A cat chases a mouse
Who is chasing?
Conv Layers as Feature Extractor
Fine-Tune for any
downstream tasks
Supervised Pre-training
Word2Vec
or
Glove
Task-specific model
unsupervised pre-training
Bert is a matured character in the comic series whereas Elmo is mischievous
Features
Embeddings
A quick comparison of pre-training
Wishlist
A model that learns a representation for a word in its context
Learned in Translation: Contextualize word Vectors (CoVe)
h_1
h_2
h_3
h_4
h_5
\sum
\alpha_{41}
\alpha_{42}
\alpha_{43}
\alpha_{44}
\alpha_{45}
c_4
s_1
Naan
s_2
transfarmar
I
enjoyed
the
film
transformers
s_3
padaththai
s_4
rasiththen
s_0
<Go>
Recall the encoder-decoder models used in MT
The entire input sentence is embedded in the hidden states of the encoder.
Can we make use of those hidden states to create a context vector?
However, \(h_1\) has encoded only the first word in the sentence, it has no idea about the rest of the words in the sentence.
How do we fixt it?
h_1
h_2
h_3
h_4
h_5
h_1
h_2
h_3
h_4
h_5
I
enjoyed
the
film
transformers
Use birectional LSTM model (biLSTM).
h_i=[\vec{h_i}:\overleftarrow{h_i}]
\rbrace
CoVe
The vector \(h_i\) is called CoVe, contextual word vector.
Capturing the context of words w.r.t a sentence is enabled by two hidden states computed in forward \((\vec{h_i})\) and backward \((\overleftarrow{h_i})\) direction.
After training the model , we can use the encoder part alone to get contextual -embeddings for a word
\overleftarrow{h_i}
\overrightarrow{h_i}
Learnable Components
A stunning story with
a
lots
of
twists and..
\begin{bmatrix}
0 \\
0\\
1\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
1 \\
0\\
0\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
0 \\
0\\
0\\
0\\
1\\
\end{bmatrix}
Task Specific
Model
Using it in the downstream task
Here, \(x\) is a sequence of words and \(v\) is the contextual embedding of those words,
x
GloVe(x)
CoVe(x)
v
However, it is still a supervised pre-training approach
GloVe
Embedding
biLSTM
v = [GloVe(x):CoVe(x)]
Wishlist
A model that learns a representation for a word in its context
Approach that make use of a large unlabelled data (that is, learn the context in an unsupervised setting)
Language modelling
Given a sequence of words/tokens (\(x_1,x_2,\cdots,x_T\)), the language model predicts the probability of the sequence
P(x_1,x_2,\cdots,x_T)=\prod \limits_{i=1}^T P(x_i|x_1,\cdots,x_{i-1})
For example
P(the,story,so,far,is,good)=P(the)P(story|the)\cdots P(good|the,story,so,far,is)
P(the,story,so,far,is,good)= 0.99
P(story,the,far,so,good,is)= 0.00001
Recall, while training the word2vec model, we computed the conditional probabilities for a word.
Is that different from the language modelling then?
Yes, there we computed the probability of a word based on the concurrency not based on the context in which it appeared.
Approximation: Bigram LMs
P(the,story,so,far,is,good)=P(the)P(story|the)P(so|story)\cdots P(good|is)
N-gram (approximation of) LMs are not a good choice for contextual embeddings as we know words can have a long-range dependency.
Therefore, language models predict a word given the entire history of words in the sentence it appeared.
The history can be either in forward direction or backward direction
P(x_1,x_2,\cdots,x_T)=\prod \limits_{i=1}^T P(x_i|x_1,\cdots,x_{i-1})
P(x_1,x_2,\cdots,x_T)=\prod \limits_{i=1}^T P(x_i|x_{i+1},\cdots,x_{T})
Forward
Backward
Bi-Directional Language modelling
The history (?) can be in both the directions (forward and backward) and often it is helpful to look into the future
I am ___ excited to watch the film as it is directed by Rajamouli
If we use only unidirectional models, then the prediction could be either not or Very.
However, in the case of bi-LMs the word not is less likely and the word Very is highly likely
Once again we can use LSTM for the modelling and we refer to that as bi-LMs hereafter.
I
went
to
the
bank
Forward LSTM
Backward LSTM
I
went
to
the
bank
Assume these two LSTMs are computing the hidden states (with initial states, (\(\overrightarrow{h_0},\overleftarrow{h_0}\))) independently on a different machine for a same input sequence
\overrightarrow{h_1}
\overrightarrow{h_2}
\overrightarrow{h_3}
\overrightarrow{h_4}
\overrightarrow{h_5}
\overleftarrow{h_1}
\overleftarrow{h_2}
\overleftarrow{h_3}
\overleftarrow{h_4}
\overleftarrow{h_5}
We can use two independent LSTM units for bi-LMs
For convenience we can group these two independent LSTM units into a block and call it a single Layer
I
went
to
the
bank
I
went
to
the
bank
\overrightarrow{h_1}
\overrightarrow{h_2}
\overrightarrow{h_3}
\overrightarrow{h_4}
\overrightarrow{h_5}
\overleftarrow{h_1}
\overleftarrow{h_2}
\overleftarrow{h_3}
\overleftarrow{h_4}
\overleftarrow{h_5}
I
went
to
the
bank
\overrightarrow{h_1}
\overrightarrow{h_2}
\overrightarrow{h_3}
\overrightarrow{h_4}
\overrightarrow{h_5}
\overleftarrow{h_1}
\overleftarrow{h_2}
\overleftarrow{h_3}
\overleftarrow{h_4}
\overleftarrow{h_5}
\overrightarrow{h_1}
\overrightarrow{h_2}
\overrightarrow{h_3}
\overrightarrow{h_4}
\overrightarrow{h_5}
\overleftarrow{h_1}
\overleftarrow{h_2}
\overleftarrow{h_3}
\overleftarrow{h_4}
\overleftarrow{h_5}
We can build as many bi-LSTM layers as we want to get deeper representations :-)
I
went
to
the
bank
\overrightarrow{h_1}
\overrightarrow{h_2}
\overrightarrow{h_3}
\overrightarrow{h_4}
\overrightarrow{h_5}
\overleftarrow{h_1}
\overleftarrow{h_2}
\overleftarrow{h_3}
\overleftarrow{h_4}
\overleftarrow{h_5}
\overrightarrow{h_1}
\overrightarrow{h_2}
\overrightarrow{h_3}
\overrightarrow{h_4}
\overrightarrow{h_5}
\overleftarrow{h_1}
\overleftarrow{h_2}
\overleftarrow{h_3}
\overleftarrow{h_4}
\overleftarrow{h_5}
We just use a single layer (instead of two) to illustrate the core idea of ELMO
Embeddings from Language Models (ELMO)
The representation for inputs is learned by a character level CNN based LM instead of word2vec.
I
went
to
the
bank
\overrightarrow{h_1}
\overrightarrow{h_2}
\overrightarrow{h_3}
\overrightarrow{h_4}
\overrightarrow{h_5}
\overleftarrow{h_1}
\overleftarrow{h_2}
\overleftarrow{h_3}
\overleftarrow{h_4}
\overleftarrow{h_5}
We just use a single layer (instead of two) to illustrate the core idea of ELMO
Embeddings from Language Models (ELMO)
The representation for inputs is learned by CNN over characters instead of word embeddings.
Concatenate hidden states of forward and backward LSTMs at time step \(t\),that is,{\(\overrightarrow{h_t}:\overleftarrow{h_t}\)}
Softmax
went
to
the
bank
\(\cdot\)
The Softmax layer is shared across time steps
The cross entropy loss is accumulated across time steps and the loss is minimized in both directions
How do we use ELMO embeddings in downstream tasks?
Learnable Components
A stunning story with
a
lots
of
twists and..
\begin{bmatrix}
0 \\
0\\
1\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
1 \\
0\\
0\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
0 \\
0\\
0\\
0\\
1\\
\end{bmatrix}
Task Specific
Model
x
\hat{x}
v
CNN over Characters
biLM
\overrightarrow{h}
\overleftarrow{h}
R_{i}=\{\hat{x_i},\overrightarrow{h_{i}},\overleftarrow{h_{i}}\}
v_i=\gamma^{task}R_i
The representation for \(i^{th}\)word:
The input to a downstream task:
What if we have multiple bi-LSTM layers?
Take a combination of representation across layers
Freeze the state of biLM layers
Learnable Components
A stunning story with
a
lots
of
twists and..
\begin{bmatrix}
0 \\
0\\
1\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
1 \\
0\\
0\\
0\\
0\\
\end{bmatrix}
\begin{bmatrix}
0 \\
0\\
0\\
0\\
1\\
\end{bmatrix}
Task Specific
Model
x
\hat{x}
v
CNN over Characters
biLM
\overrightarrow{h_{1:L}}
\overleftarrow{h_{1:L}}
R_{i}=\{\hat{x_{ij}},\overrightarrow{h_{ij}},\overleftarrow{h_{ij}}\}
How do we know that contextual embeddings are helping?
Considering the input as layer zero
j=0,1,\cdots,L
v_i=\gamma^{task}\sum \limits_{j=0}^L s_i^{task}h_{ij}
Where \(s_i^{task}\) is a task specific soft-max normalized weights.
=\{h_{ij}\}
One way is to look at whether it improves the performance of the models used in downstream tasks
For example, Question-Answering system
A mouse chases a cat
A cat chases a mouse
Who is chasing?
ELMO embeddings improved performance not only in QA task but also in named entity recognition, sentiment classification and a few more NLP tasks
Context
Question
Who is chasing?
How Contextual are Contextualized Word Representations?
Recall the examples we used earlier
he sat on the bank of the river and watched the currents
he went to the bank to check balance in his current account
Now, we expect that the embedding from ELMO for the word bank will be different based on the context
he sat on the bank of the river and watched the currents
he went to the bank to check balance in his current account
How different are these representations ?
he sat on the bank of the river and watched the currents
he went to the bank to check balance in his current account
The figure on the right shows two possible representations for the word bank appeared in different contexts
The top one gives different representations for the word bank which are closer to each other
Whereas the bottom one gives different representations for the word bank which are far from each other
The distribution of words in the embedding space of word2vec model is shown on the left.
We can clearly see that all (10K) words are distributed in all directions (it looks almost spherical)
The distribution is isotropic
What happens if we contextualize words? Does the representation still remain isotropic?
Static Embeddings
Computing cosine similarity between any two random words will result in low value (on an average)
Higher cosine similarity between any two words implies they are semantically closer.
Therefore, a set of close words forma a conic section in the sphere
Contextual Embeddings
Surprisingly, contextual embeddings are conic for all words and therefore they are anisotropic!
Moreover, the top layer captures the contextual information better than the bottom layers (the cone becomes narrow in the top layer), Ref
That is, two random words on average have high cosine similarity!
Wishlist
A model that learns a representation for a word in its context
Approach that make use of a large unlabelled data (that is, learn the context in an unsupervised setting)
Fine tuning the language model for the end task (instead of training task specific model from scratch)
Universal Language Model Fine-tuning for Text Classification
LM-Pretraining
There are three stages of training
ULMFiT
WikiText
\eta
The first stage is a usual general domain pre-training with a large text corpus.
LM contains multiple LSTM layers
The learning rate is same for all parameters in the model
LM-Pretraining
There are three stages of training
ULMFiT
WikiText
\eta
LM-Finetuning
IMDB
\eta^l_t
In the second stage, LM model is finetuned on the target vocabulary
This is required as data from target task may come from different distribution
The crucial step is to use different learning rate across layers as different layers capture different type of information
This is termed as discriminative fine tuning
LM-Pretraining
LM-Finetuning
Classifier-Finetuning
There are three stages of training
ULMFiT
WikiText
IMDB
IMDB
Labels
\eta
\eta^l_t
\eta^l_t
Two liner layers with Relu activation function are added on top the last LSTM layer
Instead of fine tuning entire model all at once, the layers gradually unfrozen from last to first layer and fine tuned.
With this approach, the performance of the model with 100 labelled samples is comparable to that of model trained with 100x samples from scratch
ULMFiT
WikiText
IMDB
IMDB
Labels
Embedding layer
Embedding layer
Embedding layer
LSTM-1
LSTM-2
LSTM-3
LSTM-1
LSTM-2
LSTM-3
LSTM-1
LSTM-2
LSTM-3
FFN + Softmax
LM-Pretraining
LM-Finetuning
\eta
\eta
\eta
\eta
\eta
\eta^4_t
\eta^3_t
\eta^2_t
\eta^1_t
\eta^5_t
\eta^4_t
\eta^3_t
\eta^2_t
\eta^1_t
ULMFiT
WikiText
Amazon Review
Amazon Review
Labels
Embedding layer
Embedding layer
Embedding layer
LSTM-1
LSTM-2
LSTM-3
LSTM-1
LSTM-2
LSTM-3
LSTM-1
LSTM-2
LSTM-3
FFN + Softmax
LM-Pretraining
LM-Finetuning
\eta
\eta
\eta
\eta
\eta
\eta^4_t
\eta^3_t
\eta^2_t
\eta^1_t
\eta^5_t
\eta^4_t
\eta^3_t
\eta^2_t
\eta^1_t
Pre-Training_and_fine-tuning
By Arun Prakash
