Preventing Overfitting

• Early Stopping
• Dropout

Preventing Overfitting - Weight Penalities

• L2 Regularization:

  $L_{\lambda }(\mathbf{w})=L(\mathbf{w})+\lambda \Vert \mathbf{w}{\Vert }_2^2$

  Also knowns as weight decay or Ridge regression

• L1 Regularization:

  $L_{\lambda }(\mathbf{w})=L(\mathbf{w})+\lambda |\mathbf{w}|$

• https://commons.wikimedia.org/wiki/File:Regularization.jpg

ReLU’s

“Vanishing Gradients” are a problem when training deep neural networks…

Logisitic function:

ReLU

Weight Initialization, Pre-Training and AutoEncoders

Momentum

nice visualizations

Batch Normalization

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Ioffe and Szegedy, 2015.

(Around 53,000 citations on Google Scholar.)

• At each layer, each input is adjusted according to:

  $${\hat{x}}^{(k)}=\frac{x^{(k)}-\text{E}[x^{(k)}]}{\sqrt{\text{Var}[x^{(k)}]}}$$

  $\text{E}[x^{(k)}]$ - The mean of $x^{(k)}$ for the batch.

  $\text{Var}[x^{(k)}]$ - The variance of $x^{(k)}$ for the batch.

• Then adjusted as:

  $$y^{(k)}={\gamma }^{(k)}{\hat{x}}^{(k)}+{\beta }^{(k)}$$

  to restore representational power.

  • ${\gamma }^{(k)}$ and ${\beta }^{(k)}$ are learnable parameters.