So far, we have learned two approaches with two different objectives  for language modelling

BERT is good at comprehension tasks like questions answering because of its bidirectional nature

GPT is good at text generation because of its unidirectional nature 

Why don't we take the best of both worlds?

A vanilla transformer architecture with GELU activation fuction.

The decoder predicts the entire sequence and computes the loss (as in the case of GPT) over the entire sequence.

The input sequence is corrupted in multiple ways: random masking, token deletion, document rotation and so on

The encoder takes in the corrupted sequence

One can think of this as a denoising-autoencoder.

 original paper

BERT, GPT,BART

BERT, GPT,BART

BERT, GPT,BART

That is, we make a copy of the pre-trained model for each task and fine-tune it on the dataset specific to that task

 Standford Sentiment Tree Bank (SST)

SNLI

LAMBADA

Predict the last word of a Long sentence

Moreover, this is not how humans adapt to different tasks once they understand the language.

We can give them a book to read and  "prompt" them to summarize it or find an answer for a specific question.

That is, we do not need "supervised fine-tuning" at all (for most of the tasks)

In a nutshell, we want a single model that learns to do multiple tasks with zero to a few examples (instead of thousands)! 

Some tasks may not have enough labelled samples

Can we adapt a pre-trained language model for downstream tasks without any explicit supervision (called zero-shot transfer)?

Note that, the prompts (or instructions) are words. Therefore,we just need to change the single task (LM) formulation

to  multi task

Yes, with a simple tweak to the input text!

For example,

Input text: I enjoyed watching the movie transformer along with my .....

Task:  summarize (or TL;DR:)

Input text: TL;DR:

Watched the transformer movie

GPT-2 with 1.5 Billion Parameters

Layers: \(4X\)

Parameters: \(10X\)

A new dataset called WebText was created by crawling outbound links (with at least 3 karma) from Reddit, excluding links to Wikipedia.

This is to ensure the diversity and quality of data 

Data of size 40 GB (about 8 million webpages)

naturally occurring demonstrations of English to French and French to English translation found throughout the WebText training set.

Uses Byte-Level BPE tokenizer

Vocabulary with 50,257 tokens

What about translation tasks?

GPT with 4 variations in architecture 

Context window size: 1024

Training objective : Causal Language Modeling (CLM)

Pre-trained

(GPT-2)

Wiki-text

text8

LAMBADA

1BW

Pre-trained

(GPT-2)

Pre-trained

(GPT-2)

Pre-trained

(GPT-2)

Let's take the pre-trained (using WebText) GPT-2  large (1.5 B) model 

 Measure the performance of the pre-trained model on various benchmarking datasets across domains such as LM, translation,summarization, reading comprehension..

The model performance increases significantly upon fine-tuning

It outperforms other language models trained on domains like Wiki and books, without using these domain-specific training datasets

Perplexity

BPC

Accuracy

PPL

Moreover, the 1.5 Billions parameters model still underfits the training set!

From the graph on the left, we can observe the trend that scaling the model and datasize further might give us some promising results on prompting.

This work established that "Large Language models are unsupervised multitask learners"

BookCorpusc(1 B )

WikiPedia (2.5 B)

Is it because of the size of the data set?

Books (0.7 B)

Is it because of the pre-training objctive?

CLM

MLM

Why is one Model performing better than the other (in all or some tasks)?

Is it because of the size of the model ( number of parameters)?

GPT

(117M)

BERT Large

(336M)

Is it because of training a model longer ?

Is it because of the way one fine-tunes the model for downstream tasks?

encoder

encoder

decoder

decoder

Scale:

• small
• medium
• large

Objective:MLM

• corruption rate
• token deletion
• span mask

Scale:

• small
• medium
• large

Scale:

• small
• medium
• large

FineTuning:

• GLUE
• SQUAD
• SuperGLUE
• WMT-14
• WMT-15
• WMT-16
• CNN/DM

hyp-params:

• num.of train steps
• learning rate scheme
• optimizer

Can we use a pre-training dataset which is as large as possible  (to understand the effect of size of unlabelled data)

Can we have the same objective for both pre-training and fine-tuning stages?

Can we have an architecture that  uses both CLM and MLM like objectives ?

Formulate all NLP problems as "Text-in" and "Text-out".

Text To Text Transfer Transformer (T5)

" translate English to Tamil: I enjoyed the movie"

" Naan padathai rasithen"

Colossal Clean Crawled Corpus (C4)

Tokenizer: SentencePiece 

Vocab Size: 32,000 

Language: Mainly English 

Number of tokens: 156 Billion (About 52 times bigger than the dataset used for pertaining BERT)

Note: We will look at the details of creating such a huge dataset in the subsequent lectures

Top 25 domains

Top 25 websites

\(I,<mask>, the ,movie\)

enjoyed

\(<go>\)

Encoder-decoder model allows us to experiment with all possible architectures and pre-training objectives

Baseline:

12 encoder and decoder stacks that is equivalent (in spirit) to \(BERT_{base}\) (with 12 layers).

Contains 220 million parameters (twice that of \(BERT_{base}\)parameters)

What sets this <w> is the pivotal role of artificial <x> (AI) in guiding the spacecraft during its <y> to the moon's surface.

What sets this mission apart is the pivotal role of artificial intelligence (AI) in guiding the spacecraft during its critical descent to the moon's surface.

<w> mission apart  <x>  intelligence <y> critical descent <z>

One advantage of this objective is that it reduces the computational cost and therefore the training time 

Batch size: 128

Context length: 512

This is just a fraction of the tokens in the entire dataset (that is, we haven't even done one epoch of training!)

BERT was trained on 137 B tokens and RoBERTa was trained on 2.2 T tokens.

However, for the given compute budget and to experiment with a variety of hyperparameters, 34B tokens is a reasonable size for the baseline model

* during fine-tuning we use same number of tokens but keep B=512, T=128

C4 Dataset

\(BERT_{base}\) Like enc-dec  with 220M parameters

Denoising objective

steps

\(\approx 34B\) tokens

GLUE

SuperGLUE

CNN/DM

SQuAD

WMT16 EnRo

steps*

\(\approx 17B\) tokens

Fine-tuning

GLUE and SuperGLUE : Collection of  tasks (datasets) to train and test natural language undetstanding.

CoLA

SST

STSB

MNLI

RTE

For text summarization. 

For Question Answering

For Translation

The evaluation metric depends on the task 

• BLEU for translation
• ROUGE for summarization
• Exact-Match, F1 for QA

report an average score (as defined by the authors of GLUE)

* a trade-off between high resource tasks (benefit from longer fine tuning) and low resource tasks (overfits)

\(\eta=0.001\)

Choose the best

model parameters for reporting the performance

 in the subsequent slides dentoes  the baseline model 

Pre-train and fine-tune the baseline model 10 times from scratch (to compute average) with different initialization and data shuffling 

Also, train the baseline model directly on the benchmarking datasets for \(2^{18}\) steps (the same number of steps used for fine-tuning)

Unsurprisingly, pre-training helps improve the model performance for small sized datasets across a variety of tasks

Supervised training with big dataset has a slight edge over pre-training as in the case of English-to-French translation

Note, however, that we cannot directly compare the performance of \(BERT_{base}\) with the basline model as the architecture, number of parameters were different and \(BERT_{base}\) was trained on \(137B\) tokens whereas baseline model was trained on 37B tokens

Whereas the baseline model achieved 80.88 on SQuAD and 84.24 on MNLI 

Therefore, for the given task, we need a way to compare different architectures 

The diagram on the right shows three architectural variants that are under study

Denoising (MLM)

CLM

Prefix - LM

The prefix language modelling objective was also used to pre-train a model under a unifying text-to-text framework [paper]

 Schematics 

Mask Patterns

Let's take a closer look at prefix LM

Dark cells: Self-attention allowed

Light cells: Self-attention not allowed

In prefix language modelling, we want the  input sequence to be attended bidirectionaly and the task output sequence autoregressively

Therefore, we can use the encoder for the input sequence and decoder for the output sequence

We can also use the decoder with the prefix-mask instead of causal mask by pre-pending the task to the input sequence

While pre-training on C4, we can take a random span of text and randomly split the text into two portions one for prefix and the other for target

we need to show that the models are equivalent in some meaningful way.

Another criterion is to use the amount of compute \(M\) (in FLOPS) required to process an input and target sequences

an encoder-decoder model with \(L\) layers in the encoder and \(L\) layers in the decoder has approximately the same number of parameters as a language model with \(2L\) layers

However, we can't use both these criteria at the same time to compare the models. Let's see why..

whereas the computational cost of both models is  the same for the given input sequence and task

How?

take it as an exercise problem :-) !  

\(P\): Number of parameters in the \(BERT_{base}\) model  (not the baseline)

\(M\): Number of FLOPS for \(L+L\) layer enc-dec model or \(L\) layer dec only model

The first row is for the baseline model which contains twice the number of parameters as compared to the \(BERT_{base}\) model

For fixed \(P\) and \(M\), Enc-dec model performs well

                                                                                                than the language model

What if we change the objective? 

denoising objective always results in better downstream task performance compared to language modeling objective.

Let's take a closer look at different unsupervised objectives 

Language

Modelling

BERT-Style

Deshuffling

Let's see an example (input,target) sequence  for each type

*this is slightly different from original BERT objective where we only predict masked tokens

** MASS-style is similar to BERT-style without random replacement of tokens

 Input text: " Thank you for inviting me to you party last week"

Overall, a BERT-style objective is better!

Both prefix-LM and BERT-style objectives are better at translation tasks!

Deshuffling objective is worse!

There are other ways of corrupting the input sequence such as, continuous masking, replace random span of text, dropping tokens

Which of these approaches for corrupting tokens is better? Let's see

Mask

Replace Tokens

Drop

All these variants perform similarly.

"Dropping corrupted tokens" leads to a slightly better performance on GLUE (because it performed well on one of the subtasks "CoLA") but worse on SuperGLUE

Overall, replacing corrupted span of tokens by a sentinel token performs well 

Well, what is a good corruption rate then? 

10%

15%

25%

50%

Corruption rate has minimal effect on performance!

But, what about the span length?

2

3

5

10

A little difference in performance (except for 10 that underperforms in two tasks)

Span length of 3 outperforms others in non-translation tasks

Let's start with the C4 (clean/filtered) dataset used for training the baseline model

Note the scale of the dataset (almost 8 times bigger than C4)

The performance degraded across all tasks despite a large increase in the scale of the dataset!

The scale of the dataset is almost 21 times smaller than C4

Despite the size, it performs well across many tasks.

Quality is important for good performance!

Let's scale it down further

These datasets are almost 42 times smaller than C4

Wikipedia:  A pre-trainig dataset used in GPT 

Wikipedia+BookCorpus:  A pre-trainig dataset used in BERT

Key take away: pre-training on in-domain unlabeled data can improve performance on downstream tasks

Fine-tuning all parameters in the model is computationally expensive

Let's consider other alternatives (to reduce the computational cost)

Add an adapter layer after FFN block in each layer 

The number of paramters in the adapter layer is typically 0.5 to 8 % of parameters in the transformer layer

During fine-tuning only the  adapter layers and layer normalization parameters are updated

We can vary the size of \(d\) and its performance on downstream tasks

Lower value of \(d\) is fine for lower-resource tasks like SQuAD (worse for other tasks)

Let us now look at another fine tuning approach called "gradual unfreezing"

Fine tuning steps: \(2^{18}\)

Number of layers: 12 each

Therefore, there will be 12 episodes of unfreezing.

Unfreezing happens for every \(\frac{2^{18}}{12}\) steps starting from top layer

Causes a slight degradation in performance

Speeds up the training process

Better results may be attainable by more carefully tuning the unfreezing schedule

It is observed that adding more quality data is helpful for better downstream performance

Why not use the datasets from all supervised learning tasks for pre-training

The unified T5 framework allows us to pre-train and/or fine-tune a single model for all tasks 

\(BERT_{base}\) Like enc-dec  with 220M parameters

Denoising objective

steps

train for

However, the dataset size for all the tasks may not be uniform

Therefore, it is important to use some sampling strategy to avoid favouring performance on one task over other

Let \(K\) be an artificial limit to the size of datasets. Say we have \(N\) tasks and original number of samples in each task is

then the probability of seeing a sample from the \(m-\)th task during training is

Sampler

\(r_m\)

Use \((r_m)^{\frac{1}{T}}\), where (\(T\)) is a temperature parameter

similar to the approach we followed for softmax

For higher values of \(T\), the distribution becomes uniform

For lower values of \(T\), the distribution becomes skewed

There is a sweet spot for the value of \(K\) that works better for some tasks

Neither increasing the value of \(K\)(size of limited dataset) nor decreasing it helps much

Temperature \(T=2\) works better for some tasks

In general, Multi-task training degrades performance than pre-train-then-fine-tune approach

\(BERT_{base}\) Like enc-dec  with 220M parameters

Denoising objective

steps

Joint Fine-tuning

steps

steps

steps

Joint Fine-tuning

Multi-task pretraining (use all datasets),

EP Sampling, \(K=2^{19}\)

Leave-one-out pretraining, (repeat for all tasks)

EP Sampling, \(K=2^{19}\)

SQuAD

CNN/DM

EP: (Examples-Proportional)

Simulates the real world where a pre-trained model is fine-tuned on a task it had not seen during pre-training

Fine-tuning after multi-task pre-training results in comparable performance to our baseline

The performance of "leave-one-out” training was only slightly worse (suggesting that a model that was trained on a variety of tasks can still adapt to new tasks)

Supervised multitask pre-training helps only in translation tasks (english pre-training does not help much)

Unsupervised pre-training+finetuning  still dominates other approaches because it was trained on \(2^{34}\) tokens!

It is well known that scaling the parameters of the model improves the performance, so we may go from \(BERT_{base}\)-like  to  \(BERT_{large}\)-like, by increasing number of layers, attention heads, \(d_{model}\), \(d_{ff}\)..

By using ensemble techniques (ensemble of 4 separately pre-trained and fine-tuned models)

Aggregate

Train the model for longer (allows the model to see more tokens, still a fraction of full dataset)

Increase the batch size (allows the model to see more tokens, for the same \(2^{19}\) time steps)

Increasing the training time (batch size) consistently improves the baseline (not a surprise)

Now, increase the model size \(2\times\) and limit the training steps to \(2t\) (that is model sees less number of tokens than the previous case)