CS6910: Fundamentals of Deep Learning

We already saw how to train this network

What about a wider network with more inputs:

where, \( \nabla w_i\) = \((f(x) − y) ∗ f(x) ∗ (1 − f(x)) ∗ \textcolor{red}{x_i}\)

\(w_1\) = \(w_1 - \eta \nabla w_1\)

\(w_{t+1} = w_t − \eta \nabla w_t\) where,

\(w_2\) = \(w_2 - \eta \nabla w_2\)

\(w_3\) = \(w_3 - \eta \nabla w_3\)

We can now calculate \(\nabla w_1\) using chain rule:

 ∗ ............... ∗ \(h_0\)

In general,

Notice that \( \nabla w_i\) is proportional to the corresponding input \(h_i−1\)

(we will use this fact later)

Backpropagation was made popular by Rumelhart et.al in 1986

However when used for really deep networks it was not very successful

In fact, till 2006 it was very hard to train very deep networks

Typically, even after a large number of epochs the training did not converge

Let us focus on the first two layers of the network \(x\) and \(h_1\)

We will first train the weights between these two layers using an unsupervised objective

Note that we are trying to reconstruct the input (\(x\)) from the hidden representation  ( \(h_1\))

We refer to this as an unsupervised objective because it does not involve the output label (\(y\)) and only uses the input data (\(x\))

Reconstruct \(x\)

We now fix the weights in layer 1 and repeat the same process with layer 2

At the end of this step, the weights in layer 2 are trained such that  captures an abstract representation of \(h_1\) 

We continue this process till the last hidden layer (i.e., the layer before the output layer) so that each successive layer captures an abstract representation of the previous layer

Is it because of better optimization?

Is it because of better regularization?

Let’s see what these two questions mean and try to answer them based on some (among many) existing studies [1,2]

Is it the case that in the absence of unsupervised pre-training we are not able to drive  (\(\mathscr{L}(\theta)\) )  to 0 even for the training data (hence poor optimization) ?

Let us see this in more detail ...

With many hills and plateaus and valleys

Given that large capacity of DNNs it is still easy to land in one of these 0 error regions

Indeed Larochelle et.al.1 show that if the last layer has large capacity then \(\mathscr{L}(\theta)\)  goes to 0 even without pretraining

However, if the capacity of the network is small, unsupervised pretraining helps

L-1 regularization: constrains most weights to be 0

L-2 regularization: prevents most weights from taking large values

Specifically, it constrains the weights to lie in regions where the characteristics of the data are captured well (as governed by the unsupervised objective)

Unsupervised objective:

We can think of this unsupervised objective as an additional constraint on the optimization problem

Supervised objective:

This unsupervised objective ensures that that the learning is not greedy w.r.t. the supervised objective (and also satisfies the unsupervised objective)

One accepted hypothesis is that pretraining leads to better weight initializations (so that the layers capture the internal characteristics of the data)

Better Activation functions

Better weight initialization strategies

Imagine if we replace the sigmoid in each layer by a simple linear transformation

\(y = (w_4 ∗ (w_3 ∗ (w_2 ∗ (w_1*x)))) \)

Then we will just learn \(y\) as a linear transformation of \(x\)

In other words we will be constrained to learning linear decision boundaries

We cannot learn arbitrary decision boundaries

But a deep non linear neural network can indeed learn such boundaries (recall Universal Approximation Theorem)

As is obvious, the sigmoid function compresses all its inputs to the range [0,1]

Since we are always interested in gradients, let us find the gradient of this function

Let us see what happens if we use sigmoid in a deep network 

While calculating  \(\nabla w_2\) at some point in the chain rule we will encounter     

What is the consequence of this ? 

To answer this question let us first understand the concept of saturated neuron ?

What would the gradient be at saturation?   

Well it would be 0 (you can see it from the plot or from the formula that we derived)

Saturated neurons thus cause the gradient to vanish.

Consider what would happen if we use sigmoid neurons and initialize the weights to very high values ?

The neurons will saturate very quickly

The gradients will vanish and the training will stall (more on this later)

Sigmoids are not zero centered

Why is this a problem??

Consider the gradient w.r.t. \(w_1\) and \(w_2\)

Note that \( \textcolor{blue}{h_{21}}\) and \( \textcolor{blue}{h_{22}}\) are between [0, 1] (i.e., they are both positive)

So if the first common term (in red) is positive (negative) then both \( \nabla w_1\) and \( \nabla w_2\) are positive (negative)

Essentially, either all the gradients at a layer are positive or all the gradients at a layer are negative 

And lastly, sigmoids are computationally expensive (because of exp (x))

Zero centered

What is the derivative of this function?

The gradient still vanishes at saturation

Also computationally expensive

Indeed it is! 

It is also called ReLU6 (6 denotes the maximum value of the ReLU)

Computationally efficient

In practice converges much faster than \(sigmoid/tanh^1\) 

Let’s see what is the derivative of ReLU(x)

\(=1            if     x > 0\)

\(=0            if     x < 0\)

Now consider the given network

What would happen if at some point a large gradient causes the bias \(b\) to be updated to a large negative value?

Not only would the output be 0 but during backpropagation even the gradient  \( \frac{\partial h_1}{\partial a_1} \)

would be zero

The weights \(w_1\), \(w_2\) and b will not get updated [∵ there will be a zero term in the chain rule] 

The neuron will now stay dead forever!! 

It is advised to initialize the bias to a positive value (0.01)

Use other variants of ReLU (as we will soon see)

Will not die (0.1x ensures that at least a small gradient will flow through)

Computationally efficient 

Close to zero centered ouputs

\( f(x) = max(αx, x)\)

α  is a parameter of the model

α   will get updated during backpropagation

Parametric ReLU

\(ae^x − 1\) ensures that at least a small gradient will flow through 

Close to zero centered outputs

Expensive (requires computation of exp(x))

Train \(k\) models with independent weights (\(w_1,w_2,\cdots w_k\)) on independently sampled data points from the original dataset.

Therefore, each model is expected to be good at predicting a subset of training samples well

During inference, the prediction is done by averaging the predictions from all the models.

About 36% of training samples are duplicates if the number of samples in all subsets are equal to the original dataset

It is often arithmetic (or geometric) mean  for regression problems

In bagging, the (sub)set of training samples seen by each of the models does not change across epochs. Therefore, the model weights get optimized for those samples

We do not have that luxury with Dropouts, especially, for deep models where the number of sub-models is exponentially high

Each sub-model is samples rarely and sees only some parts of the training data

Therefore, we want the updates to be larger

We can't set the learning rate high as the parameters are shared across the models

What is the solution?

(Mask)

Suppose that we use ReLU activation and suppose further, \( 0 < a_{21} <a_{22}\)

The weights corresponding to both \(a_{21},a_{22}\) will get updated correspondingly during backpropagation

To better reproduce the training effect in bagging, we want to update  the weights of the neuron which fit the training set well.