AI: A Moving Frontier

1958

1970

2014

1966

2023

Looked at in one way, everyone knows what intelligence is; looked at in another way, no one does. 

-Robert J. Sternberg

No, we are not being pessimistic.

This means that intelligence is complex and difficult to define.

Pursuing something that is not well-defined is challenging , but that doesn't stop us from pushing the boundaries 

A rock of any size, despite where it is in the universe, is not intelligent 

We all agree, without any definition of intelligence, that

However, small insects, like ants

, termites

are intelligent. Why?

However, their  intelligence is embedded/fixed (e.g., the weaver birds have been building the nest in the same pattern for thousands of years) and never grows (learns) with age or experience

Moreover, they do not vary across individuals (i.e., no weaver bird is superior to others in anyways)

What about Human Intelligence (HI)? 

well, we know that HI is not fixed but fluid

From ages 0 to 2, children can differentiate sounds, colors, and shapes while learning to walk and babble...

What a beautiful world it is

“a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience. ”

- From the book "The Neuroscience of Intelligence"

Where does intelligence come from?

The fundamental computational unit of the brain is the neuron.

It has billions of neurons connected to other neurons, forming trillions of connections.

The collective actions can be grouped based on their functions, such as vision, speech, hearing, motor skills, and touch.

The brain processes raw data from all senses to build the model of the world 

Therefore, unlike other species, human intelligence varies from individual to individual 

Intelligence emerges due to the  collective action of these neurons

(in 1997) An IBM machine named Deep Blue defeated a world chess champion.

Is that intelligence? 

Yes and No!

" With the same object, therefore, it is possible that one man would consider it as intelligent and another would not; the second man would have found out the rules of its behaviour."

-Alan Turing

Yes for public

No for specialists

the specific intelligence follows an explicit set of rules coded by specialists!

the machine didn't learn the rules by participating in thousands of tournaments!

So, how do we make machines to learn from data without giving any explicit set of instructions to follow?

You quickly responded "yes" without a doubt (high confidence). 

You would have seen a thousand different objects in your life. How did you distinguish this object (person) from the rest quickly ? Can you explain the process?

Is this problem easy or hard for you?

easy

"we feel we know what's going ‘on but can’t describe it properly. How could anything seem so close, yet always keep beyond our reach? "

- Marvin Minsky

Again, how did you do that? What cues did you use to distinguish a male from a female?

Though we might attempt to describe it, we do not know the exact process.

We have seen many examples from our childhood and learned to distinguish males from females

Is this problem easy or hard for you?

easy 

Is she wearing a saree?

How many ears/hands does she have?

Highly Likely (we can reason/infer from cues). All of us come with more or less similar reasoning..

Two (used general knowledge even though the picture doesn't reveal that)

Are these problems easy or hard for you?

easy 

Is this problem easy or hard for you?

Hard. 

Key takeaway: If we have an explicit set of rules to solve a problem, we can ask computers to solve it quickly.

What to learn?

The algorithm (set of rules, function,patterns..) that solves the given problem

There are many approaches. Here we focus on the approach of learning from data

How to learn?

Using machine learning models such as Artificial Neural Networks (ANN)

What is an artificial neuron in ANN?

It is a highly simplified computational model that mimics neurons in the brain

What exactly is being learned?

Let's see

male or female

Intelligence is what you use when you don’t know what to do

Model

Model

Model

Intelligence is what you use when you ____ know what to do

Model

happy or sad

positive/negative/neutral

Predict the missing word

While the models are complex and consist of billions of neurons, understanding how an individual neuron makes a decision based on a given input makes the underlying concepts clear.

The output of a decision is either yes or no

 for example, 1 means yes, 0 means No.

How does a neuron make a decision?

The magical math operations involved are "multiply and add"

We want the output to be either 0(No) or 1 (yes)

Machines can understand only numbers. 

Therefore all the inputs and outputs should be represented as numbers

We want the output to be either 0(No) or 1 (yes). 

1 if \(z>32\) else 0

Great but  

We are closer to answering that question, let's take an example

what exactly is being learned?

Suppose, you wish to buy a smart phone at a store. 

How do you decide whether to buy a particular phone or not?

* the image is for representational purposes only.

we consider a few factors (that vary across individuals) say, weight (in kg), review stars and price in thousands

Then for a particular phone these values will be \(x=[0.1,2,10]\) (that is, weight = 100 grams, stars=2, price=10k)

Then we internally weigh \((w)\) these factors to make a decision

That's what exactly the machine does! Even ChatGPT is not an exception!

Both \(x\) and\(w\) influence the decision

We can fix \(w\) (already learned) and vary \(x\) 

Let's see the decision for each configuration of \(x\) for a given \(w\) 

The three weights in \(w\) are being learned from data

For ChatGPT, it is not just three numbers, but billions of numbers (technically called parameters)

Training a neuron or neural network requires data 

The data contains examples and ground truth values (labels, annotations,..) that act as a feedback

For the second example, the model outputs 0 (reject) but the desired output is 1 (buy)

So, we have to adjust the weights (change the sliders) until it outputs 1 (buy)

Weights are adjusted until the model predicts the output label correctly for all examples

Narrow AI: AI for one particular task (emotion recognition, sentiment classification,....) using different architectures (CNN, RNN, Transformers,..)

AGI*:  Broad intelligence that solves more than one task (helpful for human interactions with chatbots like chatGPT)

The illustrative examples we have seen are discriminative, as they differentiate between categories such as male/female or happy/sad.

For decades, the majority of AI research has focused on developing algorithms to solve discriminative problems.

However, humans not only recognize (discriminate) patterns; they are also creative (generative/innovative) in both science and the arts.

Sakunthala by Ravi Varma (source)

It is very simple to be happy, but it is very difficult to be simple.

-Rabindranath Tagore

"An excited polar bear wearing swim goggles and an inner tube in a children's book illustration style with soft pastel colors. The simple character should be running on an empty, snow-dusted beach with subtle waves crashing on the shore nearby."

Artist needs to understand the description provided in natural language

If an AI is able to do that, then we call it generative AI.

As you might have guessed, the image is generated by an AI.

Today, one can generate not only images but also videos and texts. 

This has introduced a paradigm shift in the way we access information 

Stone/Iron Age 

Industrial Age

Digital  Age

Carved in Stones

Written on papers

 Digitized

 Parameterized

The Age of AI [has begun]*

*From Gatesnotes

Store and Retrieve

Store and Retrieve

Store and Retrieve

Store and Generate!

Trillions of 

Tokens

Billions of 

Parameters

Zetta FLOPS 

of Compute

GenAI

predicted word (likely)

predicted word (less likely)

If we use it, it will generate a coherent response as shown in the figure right

This is not what the user wants. We say the model doesn't understand the user's intent.

Well, we didn't train it to align it's response with the user's intent!

Using instruction tuning techniques, we align the model's responses with user intent.

The model continues to predict the next token after instruction tuning

tell

me

a

joke

about

idli

why

why

did

the

did

However, recall that

There is no doubt that GenAI is closing the gap between humans and machines.

Hallucination

Gender Bias and Stereotypes

Harmful Generation

Factual error

(hallucination is fine for creative applications (writing poems, jokes..) but not for all applications)

Additionally, the pre-training data is dominated by English.

\(\approx\)550 Billion tokens

\(\approx 1.5\) Billion tokens

For examples, taking a snapshot of the entire internet for a year (2019) and counting the number of tokens for each language will yield the following distribution.

We have about 1.1 Billion tokens for Hindi and Tamil

Therefore, these pre-trained models miss the nuances of languages other than English.

For example, even the recent GPT-4o  generates a wrong interpretation of the word "vaalarivan" in Tamil. 

One way to improve its performance is to create high-quality data in Tamil and continue pre-training the model using the new dataset. 

This is called continual pretraining. 

It is more cost-effective than pre-training

Building these datasets is both time-consuming and costly due to the involvement of humans.

Accurately recognizing the contents of rich historical documents, scripts, ancient books is essential for understanding the rich history and diverse culture of people

Building a high-performing OCR model requires high-quality annotated data.

Errors in recognition impact the performance of language models that depend on them.

शाखा कार्यालय / கிளை அனுவலகம்
चेन्ने शाखा कार्यालय - 1
चेन्ने शाखा कार्यालय - 11
मदुरै शाख कार्यालय
कोयंबट्र शाखा कार्यालय
சென்னை கிளை அலுவலகம் - 1
சென்னை கிளை அலுவலகம் - 11
மதுவர கிலை அறுவலைகள்
கோயம்புத்தூர் கிளை அலுவலகம்
mdbo-bis@bis.gov.in
cnbo1@bis.gov.in
cnbo2@bis.gov.in
cbto@bis.gov.in

Sample

Annotation

Prompt: Shakespeare, Aristotle and Plato happily talk to each other in a church

Prompt: Thiruvalluvar, Rabindranath Tagore and Bharathiyar happily talk to each other in a temple

The models were trained to understand the prompt using examples that contain a textual description of a scene (by a group of people) as shown on the right

The model may not have seen a sample that contained description similar to this.

What if we use a different model for image generation?

Though not fully correct, it could understand the prompt to some extent. 

Using more annotated data that captures culture intricacies might help the model to generate better images

Again, this requires human efforts which is time consuming and costly.

A model which is trained on multiple modalities (text, image, audio and video) is called multi-modal LLM

Again, What about the data for  Indian Context?

A long way to go!