MLF_Tutorial

Machine Learning Foundations

Tutorial - Week5

Arun Prakash A

I = \sum\limits_{i=1}^k \sigma_i u_i v_i^T

I = U \Sigma V^T

Our Mind

Let's play a game

272

268

\Sigma

V^T

268

272

Image

I am going to show you a sequence of images, one after another, that contains something.
Task : Recognise the "thing" in the images. (Note down the sequence number)
Let's go

My bad: Add +1 to the rank

What do you perceive?

Recognise the man in the picture

Rank (k =50) Approximation

Of course, we lost some resolution (details)!

But not the gross information!

Image compression

272

268

\Sigma

V^T

268

272

I = \sum\limits_{i=1}^k \sigma_i u_i v_i^T

=15

\sigma_i = 0 , i>15

272

268

Number of elements to be stored:

268*272 = 72,896

272

Original Image : (Without Compression)

Approx. Image :( or Compressed)

For \(U\) : 268*15 = 4020
For \(\Sigma\): \(k\) =15
For \( V^T\) : 15*272 = 4080
Total = 8115
~= 9 times reduction in required memory for storage

Well, but, hmm..How does it work?

Image Source: http://www.ams.org/publicoutreach/feature-column/fcarc-svd

The image is a linear combination of

\sigma_1 = 14.7

\sigma_2 = 5.22

\sigma_3= 3.31

\sigma_i= 0, i>3

It is zero by deafult!

Removes redundancy!

Tutorial - Week5

Arun Prakash A

Geometric Interpretation of SVD

SVD

I = U \Sigma V^T

Diagonal Matrix

What happens if diagonal matrices act on a set of vectors in the canonical (standard) basis?
Let us see it in \(\mathbb{R}^2\) with help of Geogebra applet :-)

Diagonal Matrices

Diagonal matrices preserves the direction of orthogonal vectors!

Why?

Similar Matrices

Geometry of SVD

A quick summary:

\(V^T\) Rotates disk D and basis \(e_1,e_2\)
\(\Sigma\) scales the rotated disk D and \(\sigma_1,\sigma_2\) are semi-major and semi-minor axis of an ellipse (hyper-ellipse)
\(U\) rotates the ellipse.

Source: Wikipedia

I = \sum\limits_{i=1}^k \sigma_i u_i v_i^T

I = U \Sigma V^T

Tutorial - Week5

Arun Prakash A

Some questions to think and solve

High Dimensional Visualization

Geoffrey Hinton

"To deal with hyper-planes in a 14-dimensional space, visualize a 3-D space and say 'fourteen' to yourself very loudly. Everyone does it.

1. Is it possible to visualize complex vectors \(x_i \in \mathbb{C}^2\)geometrically as we do for real vectors ?. Pause the video and think about it.

We need 4 dimensions to visualize a vector from \(\mathbb{C}^2\)

x_1

x_2

x = \begin{bmatrix} 1& \\ 1 \end{bmatrix}

x = \begin{bmatrix} 1+i & \\ ? \end{bmatrix}

Do complex matrices find any real world applications?

Is that just an abstract mathematical stuff?

\begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & -i & -1 & i \\ 1 & -1 & 1 & -1 \\ 1 & i & -1 & i \\ \end{bmatrix}

Discrete Fourier Transform (DFT)

Countless Applications in signal processing, Digital communication, Speech processing ...

2.Compute the inner product between two vectors \(x= \begin{bmatrix}3-2i \\-2+i\\-4 - 3i \\ \end{bmatrix} \) and \(y=\begin{bmatrix}-2+4i\\5-i\\-2i\\ \end{bmatrix}\) and verify whether they are commutative (i.e. \(x \cdot y = y \cdot x\))

\( (3+2i) \times (-2+4i)\)

\( (-2-i) \times (5-i)\)

\( (-4+3i) \times (-2i)\)

= \( -14+8i\)

= \( -11-3i\)

= \( 6+8i\)

\( -19+13i\)

\( x \cdot y = x^*y=\bar{x}^Ty\)

\( y\cdot x = y^*x=\bar{y}^Tx\)

2.Compute the inner product between two vectors \(x=[3-2i,-2+i,-4 - 3i]^T\) and \(y=[-2+4i,5-i,-2i]^T\) and verify whether they are commutative (i.e. \(x \cdot y = y \cdot x\))

\( (3-2i) \times (-2-4i)\)

\( (-2+i) \times (5+i)\)

\( (-4-3i) \times (2i)\)

= \( -14-8i\)

= \( -11+3i\)

= \( 6-8i\)

\( -19-13i\)

\( y\cdot x = y^*x=\bar{y}^Tx\)

\(\overline{x \cdot y} \)

\( y\cdot x = y^*x=\bar{y}^Tx\)

\(x \cdot y \neq y \cdot x\)

Is it always true?

3. Prove that \(x \cdot y = \overline{y \cdot x}\) where \( x \in \mathbb{C}^n\) and \( y \in \mathbb{C}^n\)

\(\overline{y \cdot x} = \overline{\bar{y}x} \)

\( = \overline{\bar{y_1}x_1}+\overline{\bar{y_2}x_2} + \cdots +\overline{\bar{y_n}x_n} \)

\( =y_1\bar{x_1}+y_2\bar{x_2}+\cdots+y_n\bar{x_n} \)

\( =\bar{x_1}y_1+\bar{x_2}y_2+\cdots+\bar{x_n}y_n \)

\( = x \cdot y\)

4. Can we use inner product to compute (cosine) angle between two complex vectors, like we do for real vectors?

No!. Not Always

\( \frac{x \cdot y}{||x|| \ ||y||} \)

Let us reason why?

But some authors prefers

\( \frac{Re(x \cdot y)}{||x|| \ ||y||} \)

More details on it : Angles in complex vector space

5. Consider the matrix \(A = \begin{bmatrix}2& 3-3i \\ 3+3i & 5 \\ \end{bmatrix} \). Find the complex eigenvector for the eigenvalue \(\lambda=8\)

R_2 = R_2+\frac{1}{2}(1+i)R_1

N[A-\lambda I] = \begin{bmatrix}-6& 3-3i \\ 3+3i & -3 \\ \end{bmatrix}

= \begin{bmatrix}-6& 3-3i \\ 0 & 0 \\ \end{bmatrix}

\begin{bmatrix}-6& 3-3i \\ 0 & 0 \\ \end{bmatrix}\begin{bmatrix}x_1 \\x_2 \\ \end{bmatrix} = \begin{bmatrix} 0 \\0 \\ \end{bmatrix}

-6x_1+(3-3i)x_2=0

-2x_1+(1-1i)x_2=0

2x_1=(1-1i)x_2

x_1 = 1

x_2 = \frac{2}{1-1i}

x_2 = \frac{2}{1-1i} = \frac{2}{1-1i} \frac{1+1i}{1+1i}

x_2 = 1+1i

\therefore x =\begin{bmatrix}1 \\ 1+1i \\ \end{bmatrix}

6. Let \(U = \begin{bmatrix} cos(t)& -sin(t) \\ sin(t) & cos(t) \end{bmatrix}\), show that the matrix \(U\) is unitary.

U = \begin{bmatrix} cos(t)& -sin(t) \\ sin(t) & cos(t) \end{bmatrix}

U^T= \begin{bmatrix} cos(t)& sin(t) \\ -sin(t) & cos(t) \end{bmatrix}

U*U^T= \begin{bmatrix} cos^2(t)+sin^2(t)& cos(t)sin(t) -sin(t)cos(t)\\ sin(t)cos(t)-cos(t)sin(t) & sin^2(t)+cos^2(t) \end{bmatrix}

U*U^T= \begin{bmatrix} 1& 0\\0 & 1 \end{bmatrix} = I

7. We know that \(U = \begin{bmatrix} cos(t)& -sin(t) \\ sin(t) & cos(t) \end{bmatrix}\),is unitary.Let us take a vector \(x \in R^2\) and see what happens when it get transformed by the \(U\).