Suppose we are given an input stream of words (sentence, document, etc.) and we are interested in learning some function of it (say,  \( \hat{y} = sentiments(words) \))

Say, we employ a machine learning algorithm (some mathematical model) for learning such a function (\( \hat{y} = f(x) \))

We first need a way of converting the input stream (or each word in the stream) to a vector \(x\) (a mathematical quantity)

This is by far AAMIR KHAN’s best one. Finest casting and terrific acting by all.

\([5.7, 1.2, 2.3, -10.2, 4.5, ..., 11.9, 20.1, -0.5, 40.7]\)

\(\uparrow\)

Model

\(\big\uparrow\)

Given a corpus, consider the set \(V\) of all unique words across all input streams (\(i.e.\), all sentences or documents)

\(V\) is called the vocabulary of the corpus (\(i.e.\), all sentences or documents)

We need a representation for every word in \(V\)

One very simple way of doing this is to use one-hot vectors of size \(|V |\)

The representation of the \(i-th\) word will have a \(1\) in the \(i-th\) position and a \(0\) in the remaining \(|V | − 1\) positions

\(V\) = [human, machine, interface, for, computer, applications, user, opinion, of, system, response, time, management, engineering, improved]

machine: 

These representations do not capture
any notion of similarity

Ideally, we would want the representations of cat and dog (both domestic animals) to be closer to each other than the representations of cat and truck

However, with \(1\)-hot representations, the Euclidean distance between any two words in the vocabulary in \(\sqrt 2\)

And the cosine similarity between any two words in the vocabulary is \(0\)

\(euclid\_ dist\)(cat, dog) = \(\sqrt 2\)

\(euclid\_ dist\)(dog, truck) = \(\sqrt2\)

\(cosine\_ sim\)(cat, dog) = \(0\)

\(cosine\_ sim\)(dog, truck) = \(0\)

Distributional similarity based representations

This leads us to the idea of cooccurrence matrix

The idea is to use the accompanying words (financial, deposits, credit) to represent bank

The context is defined as a window of \(k\) words around the terms

Let us build a co-occurrence matrix for this toy corpus with \(k = 4\)

This is also known as a word \(\times\) context matrix

You could choose the set of words
and contexts to be same or different

Each row (column) of the cooccurrence matrix gives a vectorial representation of the corresponding word (context)

Co-occurence Matrix

\(N\) \(is\) \(the\) \(total\) \(number\) \(of\) \(words\)

If  \(count(w, c) = 0,\) \(PMI(w, c) =\) \(-\infty \)

Instead use,

\(PMI_0(w, c) = PMI(w, c)\)  \(if  count(w, c) > 0\)

                \(= 0\)                           \(otherwise\) 

or

\(PPMI(w, c) = PMI(w, c)\) \(if  PMI(w, c) > 0\)

                  \(= 0\)                           \(otherwise\) 

Very sparse

Grows with the size of the vocabulary

Solution: Use dimensionality reduction (SVD)

SVD gives the best rank-\(k\) approximation of the original data \((X)\)

Discovers latent semantics in the corpus (let us examine this with the help of an example)

Each  \(\sigma_iu_i{v_i}^T \in \R^{m \times n}\)  because it is a product of a \(m \times 1\) vector with a \(1 \times n\) vector

If we truncate the sum at \(\sigma_1u_1{v_1}^T\) then we get the best rank-\(1\) approximation of \(X\) (By SVD theorem\(!\) But what does this mean\(?\) We will see on the next slide)

If we truncate the sum at \(\sigma_1u_1{v_1}^T+ \sigma_2u_2{v_2}^T\)then we get the best rank-\(2\) approximation of \(X\) and so on

Notice that \(X\) has \(m\times n\) entries

When we use the rank-\(1\) approximation. we are using only \(n + m + 1\) entries to reconstruct \([u\in  \R^m, v\in \R^n , \sigma \in \R^1 ]\)

But SVD theorem tells us that \(u_1,v_1\) and \(\sigma_1\) store the most information in \(X\) (akin to the principal components in \(X\))

Each subsequent term \((\sigma_2u_2{v_2}^T ,\) \(\sigma_3u_3{v_3}^T , . . .)\) stores less and less important information

As an analogy consider the case when we are using \(8\) bits to represent colors

The representation of very light, light, dark and very dark green would look different

But now what if we were asked to compress this into \(4\) bits\(?\) (akin to compressing \(m \times m\) values into \(m + m + 1\) values on the previous slide)

Something similar is guaranteed by SVD (retain the most important information and discover the latent similarities between words)

The \(ij-th\) entry of \(XX^T\) thus (roughly) captures the cosine similarity between \(word_i, word_j\)

\(X[i :]\)

\(X[j :]\)

\(\underbrace{\qquad \qquad \qquad }_{X}\)

\(\underbrace{\qquad \qquad \qquad }_{X^T}\)

\(\underbrace{\qquad \qquad \qquad }_{XX^T}\)

Obviously, taking the \(i-th\) row of the reconstructed matrix does not make sense because it is still high dimensional

But we saw that the reconstructed matrix \(\hat{X} = U \Sigma V^T\) discovers latent semantics and its word representations are more meaningful

Wishlist: We would want representations of words \((i, j)\) to be of smaller dimensions but still have the same similarity (dot product) as the corresponding rows of \(\hat{X}\)

\(\hat{X}\hat{X}^T =\)

\(cosine\_ sim(human, user) = 0.33\)

Notice that the dot product between the rows of the the matrix \(W_{word}=U\Sigma\) is the same as the dot product between the rows of \(\hat{X}\)

\(\hat{X}\hat{X}^T = (U \Sigma V^T )(U \Sigma V^T)^T\)

 \(= (U \Sigma V^T )(V \Sigma U^T)\)

\(= U \Sigma \Sigma^T U^T\)  \((\because V^T V=I)\)

\(= U \Sigma(U \Sigma)^T = W_{word}W_{word}^T\)

Conventionally,

\(W_{word} = U \Sigma \in  \R^{m \times k}\)

is taken as the representation of the \(m\) words in the vocabulary and

\(W_{context} = V\)

is taken as the representation of the context words

We will now see methods which directly learn word representations (these are called (direct) prediction based models)

Example: he sat on a chair

Training data:  All \(n\)-word windows in your corpus

Training data for this task is easily available (take all \(n\) word windows from the whole of wikipedia)

For ease of illustration, we will first focus on the case when \(n = 2\) (\(i.e.,\) predict second word based on first word)

\(society   (\)

\(is\)

\(is\)

\(a\)

Some sample 4 word windows from a corpus

Input: One-hot representation of the context word

Output:   There are \(|V |\) words (classes) possible and we want to predict a probability distribution over these \(|V |\) classes (multi-class classification problem)

Parameters: \(\textbf{W}_{context} \in \R^{k \times |V |}\) and \(\textbf{W}_{word} \in \R^{k \times |V |} \)

\((\)we are assuming that the set of words and context words is the same: each of size \(|V |)\)

\(sat\)

x \(\in \R^{|V |} \)

It is simply the \(i-th\) column of \(\textbf{W}_{context}\)

So when the \(i^{th}\) word is present the \(i ^{th}\) element in the one hot vector is ON and the \(i^{th}\) column of \(\textbf{W}_{context}\) gets selected

In other words, there is a one-to-one correspondence between the words and the column of \(\textbf{W}_{context}\)

More specifically, we can treat the \(i-th\) column of \(\textbf{W}_{context}\) as the representation of context \(i\)

Therefore, \(P(on|sat)\) is proportional to the dot product between \(j^{th}\) column of \(\textbf{W}_{context}\) and \(i^{th}\) row of \(\textbf{W}_{word}\)

\(P(word = i|sat)\) thus depends on the \(i^{th}\) row of \(\textbf{W}_{word}\)

We thus treat the \(i-th\) row of \(\textbf{W}_{word}\) as the representation of word \(i\)

Hope you see an analogy with SVD\(!\) (there we had a different way of learning \(\textbf{W}_{context}\) and \(\textbf{W}_{word}\) but we saw that the \(i^{th}\) row of \(\textbf{W}_{word}\) corresponded to the representation of the \(i^{th}\) word)

Now that we understood the interpretation of \(\textbf{W}_{context}\) and \(\textbf{W}_{word}\), our aim now is to learn these parameters

\(P(on|sat)=\)

We denote the context word (sat) by the index \(c\) and the correct output word (on) by the index \(w\)

For this multiclass classification problem what is an appropriate output function \((\hat{y} = f(x)) ?\)

What is an appropriate loss function\(?\)

\(\mathscr {L}(\theta)=\)\(-\)\(log\) \({\hat{y}}_w=-\)\(log\) \(P(w|c)\)

\(h = W_{context} · x_c = u_c\)

\(\hat{y}_w =\)

\(u_c\) is the column of \(W_{context}\) corresponding to context \(c\) and \(v_w\) is the row of \(W_{word}\) corresponding to context \(w\)

softmax

cross entropy

How do we train this simple feed forward neural network\(?\)

Let us consider one input-output pair \((c, w)\) and see the update rule for \(v_w\)

backpropagation

\(=\) \(-\) \(log\)

\(=\) \(-\) \((\)\(u_c . v_w\) \(-\) \(log\)

\(\nabla_{v_w}\) \(=\)

\(-(\) \(u_c\) \(-\)

\(.  u_c)\)

\(=\)  \(-\) \(u_c( 1 - \hat{y}_w)\)

If \(\hat{y}_w \rightarrow 1\) then we are already predicting the right word and \(v_w\) will not be updated

If \(\hat{y}_w \rightarrow 0\) then \(v_w\) gets updated by adding a fraction of \(u_c\) to it

This increases the cosine similarity between \(v_w\) and \(u_c\) (How\(?\) Refer to slide \(38\) of Lecture \(2\))

The training objective ensures that the cosine similarity between word \((v_w)\) and context word \((u_c)\) is maximized

The training ensures that both \(v_w\) and \(v'_w\) have a high cosine similarity with \(u_c\) and hence transitively (intuitively) ensures that \(v_w\) and  \(v'_w\) have a high cosine similarity with each other

This is only an intuition (reasonable)

Haven’t come across a formal proof for this\(!\)

So now,

\(h = \displaystyle \sum_{i=1}^{d−1} u_{c_i}\)

\([W_{context}, W_{context}]\) just means that we are stacking \(2\) copies of \(W_{context}\) matrix

The resultant product would simply be the sum of the columns corresponding to ‘sat’ and ‘he’

\( \}\)  \(sat \)

\( \}\)  \(he \)

Recall that

\(h = \displaystyle \sum_{i=1}^{d−1} u_{c_i}\)

and

\(P(on|sat, he)=\)

where ‘k’ is the index of the word ‘on’

The loss function depends on \(\{W_{word}, u_{c_1} , u_{c_2} , . . . , u_{c_{d−1}} \}\) and all these parameters will get updated during backpropogation

Try deriving the update rule for \(v_w\) now and see how it differs from the one we derived before

The denominator requires a summation over all words in the vocabulary

We will revisit this issue soon

In the simple case when there is only one \(context\) word, we will arrive at the same update rule for \(u_c\) as we did for \(v_w\) earlier

Notice that even when we have multiple context words the loss function would just be a summation of many cross entropy errors

\(\mathscr {L}(\theta)=\) \(-\) \(\displaystyle \sum_{i=1}^{d-1}\) \(log\) \(\hat{y}_{w_{i}}\)

Typically, we predict context words on both sides of the given word

Same as bag of words

The softmax function at the output is computationally expensive

Solution 1: Use negative sampling

Solution 2: Use contrastive estimation

Solution 3: Use hierarchical softmax

Let \(D\) be the set of all correct \((w, c)\) pairs in the corpus

Let \(D'\) be the set of all incorrect \((w, r)\) pairs in the corpus

\(D'\) can be constructed by randomly sampling a context word \(r\) which has never appeared with \(w\) and creating a pair \((w, r)\)

As before let \(v_w\) be the representation of the word \(w\) and \(u_c\) be the representation of the context word \(c\)

\(D = [\)(sat, on),  (sat, a),  (sat,  chair),  (on, a), (on,chair), (a,chair), (on,sat),   (a,    sat), (chair,sat),  (a,    on), (chair, on), (chair, a) \(]\)

\(D' = [\)(sat,   oxygen), (sat, magic),   (chair, sad), (chair, walking)\(]\)

Let us model this probability by

\(p(z = 1|w, c) = \sigma(u_c^T  v_w)\)

Considering all \((w, c) \in D\), we are interested in

\(= \frac{1}{1   + e^{-u_c^T v_w}}\)

\( maximize \displaystyle \prod_{(w,c) \in D} p(z = 1|w, c)\)

where \(\theta\) is the word representation \((v_w)\) and context representation \((u_c)\) for all words in our corpus

Again we model this as

\(p(z = 0|w, r) = 1 − \sigma(u_r^T v_w)\)

\(=\) \(1-\) \(\frac{1}{1   + e^{-u_r^T v_w}}\)

\(=\) \(\frac{1}{1   + e^{u_r^T v_w}}\)  \(=\) \(\sigma(-u_r^T v_w)\)

Considering all \((w, r) \in D'\) , we are interested in

\( maximize \displaystyle \prod_{(w,r) \in D'} p(z = 0|w, r)\)

\(\theta\)

\( \displaystyle \prod_{(w,r) \in D^{'}} p(z = 0|w, r)\)

\( \displaystyle \prod_{(w,r) \in D^{'}} (1-p(z = 1|w, r))\)

\(+\)  \( \displaystyle \sum_{(w,r) \in D^{'}}\) \(log\) \((1-p(z = 1|w, r))\)

\( \frac{1}{1   + e^{-v_c^T v_w}}\)

\(+\)  \( \displaystyle \sum_{(w,r) \in D^{'}}\) \(log\)

\( \frac{1}{1   + e^{v_r^T v_w}}\)

\(+\)  \( \displaystyle \sum_{(w,r) \in D^{'}}\) \(log\) \(\sigma(-v_r^T v_w)\)

where \(σ(x)\) \(=\)  \(\frac{1}{1+e^{-x}}\)

The size of \(D'\) is thus \(k\) times the size of \(D\)

The random context word is drawn from a modified unigram distribution

\(r ∼ p(r)^{\frac{3}{4}}\)

\(r ∼\) \(\frac{count(r)^{\frac{3}{4}}}{N}\)

\(N\) \(=\) total number of words in the corpus

We would like \(s\) to be greater than \(s_c\)

So we can maximize \(s − (s_c + m)\)

Okay, so let us try to maximize \(s − s_c\)

But we would like the difference to be at least \(m\)

What if \(s > s_c + m\) \((\) \(don’t\) \(do\) \(any\) \(thing\) \()\)

maximize   \(max(0, s − (s_c + m))\)

There exists a unique path from the root node to a leaf node.

Let \(l(w_1), l(w_2), . . ., l(w_p)\) be the nodes on the path from root to \(w\)

Let \(π(w)\) be a binary vector such that:

\(π(w)_k = 1\)   path branches left at node \(l(w_k)\)

\(= 0\)   otherwise

Finally each internal node is associated with a vector \(u_i\)

So the parameters of the module are \(\textbf W_{context}\) and \(u_1, u_2, . . . , u_v\) (in effect, we have the same number of parameters as before

\(u_ V\)

\(u_ 1\)

\(u_ 2\)

\(π(on)_1 = 1\)

\(π(on)_2 = 0\)

\(π(on)_3 = 0\)

We model this probability as

\(p(w|v_c) =\displaystyle \prod_{k} (π(w_k)|v_c)\)

For example

\(P(on|v_{sat}) = P(π(on)_1 = 1|v_{sat})\)

               \(* P(π(on)_2 = 0|v_{sat})\)

               \(* P(π(on)_3 = 0|v_{sat})\)

In effect, we are saying that the probability of predicting a word is the same as predicting the correct unique path from the root node to that word

\(P(π(on)_i = 1) =\)

\(P(π(on)_i = 0) = 1 − P(π(on)_i = 1)\)

\(P(π(on)_i = 0) =\)

The above model ensures that the repres-entation of a context  word \(v_c\) will have  a high(low) similarity with the representation of the node \(u_i\) if \(u_i\) appears  and  the   path branches to the left(right) at \(u_i\)

Again, transitively the representations of contexts which appear with the same words will have high similarity

Note that  \(p(w|v_c)\) can now be computed using  \(|π(w)|\) computations instead of \(|V |\) required by softmax

How do we construct the binary tree\(?\)

Turns out that even a random arrangement of the words on leaf nodes does well in practice