JMU CS 445: Training Neural Networks

Preventing Overfitting

  • Early Stopping
  • Dropout

Preventing Overfitting - Weight Penalties

  • L2 Regularization:

    Lλ(w)=L(w)+λ∥w∥22L_\lambda(\mathbf{w}) = L(\mathbf{w}) + \lambda \|\mathbf{w}\|_2^2

    Also known as weight decay or Ridge regression

  • L1 Regularization:

    Lλ(w)=L(w)+λ∣w∣L_\lambda(\mathbf{w}) = L(\mathbf{w}) + \lambda |\mathbf{w}|

ReLU's

"Vanishing Gradients" are a problem when training deep neural networks...

Logistic function:

The logistic function from z = -10 to 10, an S curve from 0 to 1, and below it its derivative, a bump that peaks at 0.25 at z = 0 and is near zero elsewhere.

ReLU:

The ReLU function from z = -10 to 10, zero for negative z and equal to z for positive z, and below it its derivative, 0 for negative z and 1 for positive z.

Weight Initialization, Pre-Training and AutoEncoders

Momentum

nice visualizations

Batch Normalization

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Ioffe and Szegedy, 2015.

(Around 70,000 citations on Google Scholar.)

  • At each layer, each input is adjusted according to:

    x^(k)=x(k)−E[x(k)]Var[x(k)]\hat{x}^{(k)} = \frac{x^{(k)} - \text{E}[x^{(k)}]}{\sqrt{\text{Var}[x^{(k)}]}}

    E[x(k)]\text{E}[x^{(k)}] - The mean of x(k)x^{(k)} for the batch.

    Var[x(k)]\text{Var}[x^{(k)}] - The variance of x(k)x^{(k)} for the batch.

Batch Normalization (continued)

  • Each normalized input is then adjusted as:

    y(k)=γ(k)x^(k)+β(k)y^{(k)} = \gamma^{(k)}\hat{x}^{(k)} + \beta^{(k)}

    to restore representational power.

    • γ(k)\gamma^{(k)} and β(k)\beta^{(k)} are learnable parameters.

Marp version of regularization_old.md (pandoc format).