GPT-2 (1.5 Billion parameters) and T5 (10 Billion parameters) were trained on 19 Billion and  175 Billion tokens, respectively. 

What are all the aspects (duplication, toxicity,bias) that we need to consider while building a larger dataset?

The other factor that motivated scaling the data size was the study on Scaling Laws

(published a few months after T5) 

Given the number of non-embedding parameters \(N\) and the size of the dataset \(D\) (in tokens), the test loss 

where,

\(N_c = 8.8 \times 10^{13}\) : 88 Trillion parameters

\(D_c=5.4 \times 10^{13}\) : 54 Trillion tokens

\(\alpha_N = 0.076 \)

\(\alpha_D = 0.095 \)

The study on T5 demonstrated that the model overfits if the data is repeated for many epochs during training. This suggests that we should scale the data size

Suppose we have a model with 100 million (non-embedding) parameters

and a dataset with 10 million tokens

What will be the training loss

Of course, the model will drive the training loss to zero!

However, the test loss changes according to the scaling law

What happens if we increase the size of the dataset? Let's see

What happens if we increase the size of the dataset? Let's see

Click on the slider to increase the value of D by 10 million tokens (or use right/left arrows in the keyboard)

Increasing the number of tokens reduces the model's test loss 

This motivated the community to scale both the model and data size [Ref]

Fruit Fly

Honey Bee

Mouse

Cat

Brain

#  Synapses

Transformer

GPT-2

Megatron LM

GPT-3

GShard

Switch

BookCorpus

Wikipedia

WebText (Closed)

RealNews

C4

While data has been growing......

Open source data is good enough

 The Pile

ROOTS

Falcon

RedPajama

DOLMA

Scale

Quality

Diversity

LLM

Detailed information about the data-cleaning pipeline is available publicly for the following datasets

Often, datasets used in pre-training large models lack proper documentation. Technical reports typically only provide very basic information

For example, MassiveWeb used in training Gopher model uses a custom HTML scraper using Googles SafeSearch signal for filtering

In this lecture, we attempt to give you a comprehensive idea about various data processing pipelines.

One can write a custom HTML scraper or use Common Crawl (CC) for fetching the data 

Each month they release a snapshot of the web that contains about 3 to 5 billion web pages 

the raw data contains duplicate, toxic, low-quality content and also personally identifiable information

Garbage in - Garbage out (GIGO)

Recall the saying

So it is important to clean the raw text  

Well, the web contains more than 100 petabytes of data. 

Let us understand the data format of the CC snapshots 

Therefore, it is a challenging task  to build an automatic pipeline that takes in the raw text and outputs clean text

The pipeline takes in the raw data in any one of these formats and produces  clean text

To build a large dataset, researchers often take CC snapshots of multiple months or years

It is also challenging to directly measure how clean the output text is.

To get high quality text from the raw data, the pipeline requires mechanisms 

to detect toxicity level (bad words, hate speech, stereotypes,..) of the content in the pages

to detect the language of the web pages 

to detect quality of the content in the  pages 

to deduplicate the content (line, paragraph and document level)

Exact duplication 

The orbit of Aditya-L1 spacecraft is a periodic Halo orbit

The orbit of Aditya-L1 spacecraft is a periodic Halo orbit

webpage-1

webpage-2

Fuzzy (near) duplication

The orbit of Aditya-L1 spacecraft is a periodic Halo orbit which is located roughly 1.5 million km from earth on the continuously moving Sun  

The orbit of Aditya-L1 spacecraft is a periodic Halo orbit . The orbit is located roughly 1.5 million km from earth on the continuously moving Sun  

Exact deduplication  Algorithm

Fuzzy deduplication  Algorithm

Suffix array

Language Detection

hi

ta

en

Deduplication

(using hashing)

Deduplication

(using hashing)

Deduplication

(using hashing)

It satisfies only two requirements

Uses line level deduplication

The filtered text could still contain toxic, low-quality and PII

This is also helpful for low-resource languages as datasets like Bookcorpus and Wikipedia are only for English

Even if we wish to use  Wikipedia dumps for other languages, the size won't be sufficient 

For example, Hindi has only 1.2 million pages,Tamil has only 0.5 million pages where as English has 59 million pages [source]

Therefore, we need to use CC dumps (years of dumps)

This is a compute-intensive task and requires massive parallelization with carefully written code to execute them

Divide each webpage into three portions: head, middle and tail . Obtain quality score for each portion using a pre-trained language model on a targeted domain (say, Wikipedia)

ML Classifier

ML Classifier

ML Classifier

quality score

quality score

quality score

Targeted Quality filtering

It satisfies three requirements

By following this pipeline, they have extracted documents for 130 languages (that has at least 1000 documents)

Recall, Tamil has 0.5 million Wikipedia pages, whereas by using  CCNet pipeline on Feb 2019 snapshot of common crawl, one could get 0.9 million documents for the language (Note, there may be a overlap)

\(\approx\)550 Billion tokens

\(\approx 1.5\) Billion tokens

It is an extension of CCNet with additional filtering heuristics. 

We can get diverse data such as news, stories, webtext by domain filtering

They released the clean dataset (named C4) in the public domain

Menu

Lemon

Introduction

The lemon, Citrus Limon (l.) Osbeck, is a species of small evergreen tree in the flowering plant family rutaceae. The tree's ellipsoidal yellow fruit is used for culinary and non-culinary purposes throughout the world, primarily for its juice, which has both culinary and cleaning uses. The juice of the lemon is about 5% to 6% citric acid, with a ph of around 2.2, giving it a sour taste.

Article

The origin of the lemon is unknown, though lemons are thought to have first grown in Assam (a region in northeast India), northern Burma or China. A genomic study of the lemon indicated it was a hybrid between bitter orange (sour orange) and citron.

Please enable JavaScript to use our site.

Home

Products

Shipping

Contact

FAQ

Dried Lemons, $3.59/pound

Organic dried lemons from our farm in California.

Lemons are harvested and sun-dried for maximum flavor.

Good in soups and on popcorn.

The lemon, Citrus Limon (l.) Osbeck, is a species of small evergreen tree in the flowering plant family rutaceae. The tree's ellipsoidal yellow fruit is used for culinary and non-culinary purposes throughout the world, primarily for its juice, which has both culinary and cleaning uses. The juice of the lemon is about 5% to 6% citric acid, with a ph of around 2.2, giving it a sour taste

Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Curabitur in tempus quam. In mollis et ante at consectetur.

Aliquam erat volutpat. Donec at lacinia est.

Duis semper, magna tempor interdum suscipit, ante elit molestie urna, eget efficitur risus nunc ac elit.

Fusce quis blandit lectus. Mauris at mauris a turpis tristique lacinia at nec ante.

Aenean in scelerisque tellus, a efficitur ipsum.

Integer justo enim, ornare vitae sem non, mollis fermentum lectus.

Mauris ultrices nisl at libero porta sodales in ac orci.

function Ball(r){

this.radius = r;

this.area = pi * r ** 2;

this.show = function()

{ drawCircle(r);

}

}

has a lot of gibberish

boiler-plate text like menus

offensive language, placeholder text, source code, etc.

Make clean corpus by filtering out low quality text that is suitable for NLP tasks

Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Curabitur in tempus quam. In mollis et ante at consectetur.

Aliquam erat volutpat. Donec at lacinia est.

Duis semper, magna tempor interdum suscipit, ante elit molestie urna, eget efficitur risus nunc ac elit.

Fusce quis blandit lectus. Mauris at mauris a turpis tristique lacinia at nec ante.

Aenean in scelerisque tellus, a efficitur ipsum.

Integer justo enim, ornare vitae sem non, mollis fermentum lectus.

Mauris ultrices nisl at libero porta sodales in ac orci.

function Ball(r){

this.radius = r;

this.area = pi * r ** 2;

this.show = function()

{ drawCircle(r);

}

}

Remove any page that contains "lorem ipsum" or curly bracket 

In general, all pre-processing data pipelines follow CCNet pipeline with modifications in the components of the pipeline

For example, pile-CC uses the WARC files (instead of WET) which contain raw HTML pages then applies filtering (using jusText) to extract text content

jusText

(Extract text data)

.WARC

Language Detection

(Pycld2)

Deduplication and Targeted Quality filtering

It uses fuzzy deduplication over exact deduplication algorithm as in C4

Pile-CC

(227 GiB)

Significant duplicates in CC are exact duplicates [Ref]. Therefore, one can use fuzzy deduplication first followed by exact deduplication.

50% of CC was removed after language identification

RW-Raw

RW-Raw

RW-Filtered

It used both fuzzy and exact deduplication techniques.

What is the impact of having duplicate content?

In total, this stringent approach removes almost 90% of data from a single snapshot of the common crawl

RW-Raw

RW-Filtered

So far we have seen that the pre-processing pipeline takes in different forms of text data. Why not use Audio (using ASR), Scanned documents (using OCR)?

Using these  modalities are helpful for low-resource languages

Language Detection: 

• INDICLID, CLD3, NLLB

Scraping: 

• Trafilatura 

Deduplication: 

• Fuzzy (MinHash)

Not only in C4 but also in other datasets such as RealNews [Ref]

Therefore, the natural question is, Does duplicate content degrade the performance of the model? 

Including the Pile: "We found the Pile was particularly full of duplicate documents, and advise future researchers using the Pile to perform additional de-duplication processing" [Ref]*

* however, the Pythia suite of models found that deduplicating The Pile had a limited impact on zero-shot performance

For instance, a sequence that is present 10 times in the training data is on average generated 1000x more often than a sequence that is present only once [Ref]

for a 1B parameters model, a hundred duplicates are harmful; at 175B, even a few duplicates could have a disproportionate effect [Ref]

Pre-training Datasets

Monolingual

Multilingual

We can also group based on the diversity of data sources (web,books,wiki,code..)

en

en,gu,hi,sa,ta,te..

For convenience, the source code of 300+ programming languages can be considered as monolingual (english) datasets

code

Some publically released pre-training datasets (for ex, BookCorpus) are no longer available

So researchers build their own versions (say, OpenWebText) by scraping the web by following similar steps in the original paper.

Some datasets are private, for example, webtext

One way to compare the size of the datasets is to use the (uncompressed) storage memory

The other useful way is to compare the size of the datasets in number of tokens (related to scaling law)

Datasets such as The Pile, and DOLMA emphasize diversity in the pre-training data. 

Datasets such as RefinedWeb suggest that one could get good performance with web only data 

~156 Billion

No

Webpage

~170 Billion

Yes

22 sources

> 1 Trillion

Yes

380 Programing languages

5 Trillion (600B in public)

No

Webpage

 1.2/30 Trillion

Yes

Webpage, Books, Arxiv, Wiki, StackExch

3 Trillion

Yes

Webpage, Books,  Wiki, The Stack, STEM

~418 Billion

No

Webpage

~341 Billion

Yes

natural and programming languages

251 Billion

Yes

English

English

Code

English

English/Multi

English

Multi

Multi

Multi

RedPajama-V1* (2022)

DOLMA (2024)

Observe the increase in token counts for each year 

Note, however, that the recent RefinedWeb pipeline demonstrates that sourcing a large scale, high quality data from web alone is sufficient to get a better performance[Ref] 

Suppose we decided to train a model for 15 million training steps  (\(2^{16}\) tokens per training step), resulting in 1 Trillion tokens

Then we take these \(2^{16}\) tokens by sampling from all the available data sources according to the pre-defined distribution

GPT-3

PaLM

LLaMA

Almost all LLM models predominately used web data (from The Pile, C4, CC, Custom scraping)

WiKi and Books are the next common source

Models like ALPHA CODE uses 100% code data

Models like LaMDA and PALM gave more weightage to conversation data

Some of these models were trained on multilingual datasets, as shown in the next slide

M-L: Multi-Lingual

H: Heuristics

C: Classifiers