\( \)
\(\nabla _ {a_{k-1}} \mathscr {L} (\theta) = \nabla _ {h_{k-1}} \mathscr {L} (\theta) \odot [...,g' (a_{k-1,j}),...];\)
\(\nabla _ {W_k} \mathscr {L} (\theta) = \nabla _ {a_k} \mathscr {L} (\theta) h_{k-1}^T ;\)
\(\nabla _ {b_k} \mathscr {L} (\theta) = \nabla _ {a_k} \mathscr {L} (\theta) ;\)
\(f(x) = max(0, x) \)
\(f(x) = max(0, x ) − max(0, x − 6)\)
\(f(x)= max(0, x)\)
\(w_1x_1 + w_2x_2 + b < 0 [if b << 0]\)
\(f(x) = max(0.1x,x) \)
Sampling with replacement