Maximum Likelihood Learning

Bayesian Learning

• In Bayesian learning, we use data to update a probability distribution over our model parameters \(\theta\) given our training data \(\mathcal{D} = (\mathbf{x}_1, \mathbf{x}_2, ... \mathbf{x}_n)\):

• \(P(\theta \mid \mathcal{D}) = \frac{ P( \mathcal{D} \mid \theta) P(\theta)} {P(\mathcal{D})}\)

• The goal here is not to find the single best fit for our parameters. Instead we maintain a probability distribution over our parameters.

Maximum a Posteriori Learning

• An alternative is to attempt to find the single most probable set of parameters given our data and our prior:

• \[\theta_{MAP} = \text{argmax}_\theta P(\mathcal{D} \mid \theta) P(\theta)\]

Maximum Likelihood Learning

• An even simpler alternative is to just pick the parameters that make the data most probable: \[\theta_{ML} = \text{argmax}_\theta P(\mathcal{D} \mid \theta)\]

• \(P(\mathcal{D} \mid \theta)\) is commonly referred to as likelihood: \[\mathcal{L}(\theta) = P(\mathcal{D} \mid \theta) = \prod_{\mathbf{x}_i \in \mathcal{D}} P(\mathbf{x}_i \mid \theta)\] (The likelihood isn't really a probability distribution... We know what the data is, so the probability of the data is 1. It is a measure of how probable (likely) that data is given our model parameters.)

• Numerically, we are usually better off working with the log likelihood: \[\mathcal{LL}(\theta) = \sum_{\mathbf{x}_i \in \mathcal{D}} \log P(\mathbf{x}_i \mid \theta)\]

Turning our single layer network into a probability distribution...

• This is the logistic or sigmoid function: \[\sigma(z) = \frac{1}{1 + e^{-z}}\]

• It looks like this:

  

• If we let \(z = \mathbf{w}^T\mathbf{x} + b\), we end up with: \[ \begin{split} P(y = 1 \mid \mathbf{x}, \mathbf{w}) &= \sigma( \mathbf{w}^T\mathbf{x} + b ) = \frac{1}{1 + e^{-(\mathbf{w}^T\mathbf{x }+ b)}}\\ P(y = 0 \mid \mathbf{x}, \mathbf{w}) &= 1 - \sigma( \mathbf{w}^T\mathbf{x} + b ) \end{split} \]

Aside: Derivative of the Sigmoid/Logistic Function

The logistic function has a simple derivative:

\[\sigma'(x) = \sigma(x)(1 - \sigma(x))\]

One Weird Trick...

• We want a likelihood function for our binary classifier. It could look like this: \[\mathcal{L}(\mathbf{w}) = \prod_{i=1}^n \begin{cases} P(y = 1 \mid \mathbf{x_i}, \mathbf{w}), \text{if } y_i = 1 \\ P(y = 0 \mid \mathbf{x_i}, \mathbf{w}), \text{if } y_i = 0 \\ \end{cases} \]

• This is much nicer to work with: \[\mathcal{L}(\mathbf{w}) = \prod_{i=1}^n P(y = 1 \mid \mathbf{x_i}, \mathbf{w})^{y_i} \times P(y = 0 \mid \mathbf{x_i}, \mathbf{w})^{1-y_i} \] (remember... \(x^1 = x, x^0 = 1\))

• This makes the log likelihood: \[\mathcal{LL}(\mathbf{w}) = \sum_{i=1}^n y_i \log P(y = 1 \mid \mathbf{x_i}, \mathbf{w}) + (1 - y_i) \log P(y = 0 \mid \mathbf{x_i}, \mathbf{w}) \] (remember... \(\log_b(x^y) = y\log_b(x)\))

• The goal is to maximize the log likelihood, which is the same as minimizing the negative log likelihood: \[-\mathcal{LL}(\mathbf{w}) = -\sum_{i=1}^n y_i \log P(y = 1 \mid \mathbf{x_i}, \mathbf{w}) + (1 - y_i) \log P(y = 0 \mid \mathbf{x_i}, \mathbf{w}) \]

Maximum Likelihood for Logistic Regression:

• This loss function is usually called cross-entropy. \[-\mathcal{LL}(\mathbf{w}) = -\sum_{i=1}^n y_i \log(\sigma( \mathbf{w}^T\mathbf{x} + b )) + (1 - y_i) \log (1 - \sigma( \mathbf{w}^T\mathbf{x} + b )) \]

• Quiz: What is the cross-entropy loss for \(\sigma( \mathbf{w}^T\mathbf{x} + b ) = 1\) and \(y_i = 0\)? \(y_i = 1\)?

  Remember... \(\log(1) = 0, ~~~ \log(0) = -\infty\)

Why not just use MSE?

• Zero and one are numbers, so there is no reason we couldn't use the loss function we used for linear regression: \[E(\mathbf{w}) = \sum_{i=1}^n (y_i - \sigma(\mathbf{w}^T\mathbf{x} + b))^2\]

• Here is how the two loss functions compare: