Maximum likelihood estimation

Where loss functions actually come from. Pick a probability model, ask which parameters make the data you saw most probable, and out fall the sample mean, the fraction of heads, and the cross-entropy loss you have been using since chapter 11.

1. Why we need this: where do loss functions come from?

So far each model came with a loss that was just handed to you. Linear regression used squared error. Logistic regression used cross-entropy, justified by "it's convex and the gradient is clean". Why those? Would $|y - \hat y|^3$ be just as good?

Maximum likelihood estimation (MLE) is the principle behind those choices. It goes in two steps:

1. Say what you believe about how the data was generated, as a probability distribution with unknown parameters $\theta$: "each flip is heads with probability $\phi$", "grades are bell-shaped around some $\mu$".
2. Choose the $\theta$ under which the data you actually observed is most probable.

Do that and the loss isn't a choice any more. It's derived. By the end of this chapter, cross-entropy will come out of a coin-flip model with no extra assumptions (§8).

2. The four distributions we'll use

The first two are discrete (they give probabilities of outcomes); the normal distributions are continuous (they give probability densities).

Two notes on notation. In the categorical row, the outcome is stored as a one-hot vector: "green" on a red/green/blue/magenta die is $[0,1,0,0]^\top$, so exactly one $Y_k$ is 1. In the normal row, $\sigma^2$ is the variance and $\sigma$ the standard deviation. For the multivariate normal, $|\Sigma|$ is the determinant of the covariance matrix and $\Sigma^{-1}$ its inverse; in 2-D the bell curve becomes a "hat".

3. Likelihood: how probable is the data, given the parameters?

Why can we multiply? Because we assume the data is i.i.d.: independent (one flip doesn't affect the next) and identically distributed (every flip uses the same $\theta$). For independent events, the joint probability is the product. Without that assumption you'd be stuck with each outcome depending on all the previous ones, each with its own parameters:

which you couldn't estimate from one dataset.

Three worked likelihoods

4. Maximum likelihood: flip the question around

A single likelihood value like $3.42\times10^{-6}$ means almost nothing on its own. Comparing two values does: it told us $(85, 10^2)$ beats $(80, 5^2)$. MLE takes that to its conclusion and searches over all parameter values for the best one:

To stress that we're now varying $\theta$ with the data held fixed, it's written as the likelihood function $L(\theta;\D) = p(\D\mid\theta)$, often just $L(\theta)$. It is the same formula as before. The only thing that changed is which symbol we treat as the variable.

5. Take the log, then negate it

In practice we never maximise $L$ directly. We use the log-likelihood

for two reasons:

Why is it allowed? $\log$ is monotonically increasing: if $L(\theta_B) > L(\theta_A)$ then $\log L(\theta_B) > \log L(\theta_A)$. It changes the values but never the order, so the $\theta$ at the top is the same:

Then negate. We've built all our tools around minimising (gradient descent), so flip the sign: $\argmax_\theta \ell(\theta) = \argmin_\theta\,[-\ell(\theta)]$. The negative log-likelihood (NLL) is now an ordinary loss to minimise.

6. MLE for a coin: the fraction of heads

A notation trick first. With specific data we could write each factor as $\phi$ or $1-\phi$ by inspection. With a general $y^{(i)}$ we need one expression that picks the right one automatically. Use the indicator $\mathbb 1(\cdot)$, which is 1 when its condition is true and 0 otherwise, as an exponent:

If $y^{(i)} = 1$ this is $\phi^1(1-\phi)^0 = \phi$. If $y^{(i)} = 0$ it's $\phi^0(1-\phi)^1 = 1-\phi$. Anything to the power 0 is 1, so the factor that doesn't apply drops out.

Derivation

Suppose $\D$ has $N_H$ heads and $N_T$ tails ($N = N_H + N_T$).

That's the answer you'd have guessed, the fraction of flips that came up heads. MLE agrees with intuition, and the same machinery keeps working when intuition runs out.

7. MLE for a die: the trick the notes held back

The same indicator trick, with one-hot data. For data point $i$, exactly one $y_k^{(i)}$ is 1:

For "green" $= [0,1,0,0]^\top$, the inner product is $\phi_1^0\phi_2^1\phi_3^0\phi_4^0 = \phi_2$. Every non-matching side contributes a factor of 1. Taking logs and swapping the order of the two sums:

where $N_k$ is how many times side $k$ came up.

The notes stop here: "we're going to hold off… as it involves one more trick related to the fact that $\sum_k\phi_k = 1$". Here's that trick.

8. MLE for a Gaussian: the mean, and the variance

Data $x^{(1)},\dots,x^{(N)}$, model $\mathcal N(\mu,\sigma^2)$. Log of each density:

so summing over $i$,

The log has removed the exponential, as promised.

going deeper $\hat\sigma^2$ divides by $N$. On average it slightly underestimates the true variance, because $\hat\mu$ was fitted to the same data and sits closer to it than the true $\mu$ does. The "sample variance" in statistics divides by $N-1$ to correct this (108.33 instead of 72.22 for the grades). MLE gives the $N$ version.

9. The payoff: logistic regression is MLE

So far we've modelled one variable on its own. For prediction we want the distribution of the label given the input, $p(y\mid\mathbf x,\theta)$. The recipe: collect i.i.d. pairs $(\mathbf x^{(i)}, y^{(i)})$, choose a form for $p(y\mid\mathbf x,\theta)$, and maximise

Binary

Logistic regression says $p(y = 1\mid\mathbf x) = \sigma(\theta^\top\mathbf x) = \frac{1}{1+e^{-\theta^\top\mathbf x}}$. Write $\phi^{(i)} = \sigma(\theta^\top\mathbf x^{(i)})$. Each label is now a coin flip whose bias depends on its input. So the likelihood is the Bernoulli one from §6, with a different $\phi$ per point:

Now make three substitutions, each true by definition: $\mathbb 1(y^{(i)}=1) = y^{(i)}$ (since $y\in\{0,1\}$), $\mathbb 1(y^{(i)}=0) = 1 - y^{(i)}$, and $\phi^{(i)} = \hat y^{(i)}$:

Multi-class

With softmax, $\phi_k^{(i)} = \frac{\exp(\theta_k^\top\mathbf x^{(i)})}{\sum_{l=1}^K\exp(\theta_l^\top\mathbf x^{(i)})}$: each label is a die roll whose weights depend on the input. That gives the categorical likelihood from §7:

which is the multi-class cross-entropy, again.

And linear regression? going deeper

Assume $y = \theta^\top\mathbf x + \varepsilon$ with noise $\varepsilon\sim\mathcal N(0,\sigma^2)$, so $p(y\mid\mathbf x) = \mathcal N(\theta^\top\mathbf x,\sigma^2)$. Then

The second term doesn't involve $\theta$ and $\frac{1}{2\sigma^2}$ is a positive constant, so minimising the NLL over $\theta$ is minimising the sum of squared errors. Squared error is the NLL of Gaussian noise.

10. Recap

1. Likelihood $p(\D\mid\theta) = \prod_i p(y^{(i)}\mid\theta)$ under i.i.d. As a function of $\theta$ it's $L(\theta)$. It is not a distribution over $\theta$.
2. MLE: $\hat\theta = \argmax_\theta L(\theta) = \argmax_\theta \ell(\theta) = \argmin_\theta -\ell(\theta)$. Logs avoid underflow and simplify the calculus, and they're safe because log is monotonic.
3. Recipe: likelihood → NLL → derivative → set to 0 or run gradient descent.
4. Indicator-exponent trick: $\phi^{\mathbb 1(y=1)}(1-\phi)^{\mathbb 1(y=0)}$ and $\prod_k\phi_k^{\mathbb 1(y_k=1)}$.
5. Results: Bernoulli $\hat\phi = N_H/N$; categorical $\hat\phi_k = N_k/N$ (Lagrange multiplier, $\lambda = N$); Gaussian $\hat\mu = $ mean and $\hat\sigma^2 = \frac1N\sum(x-\hat\mu)^2$.
6. Conditional MLE with a Bernoulli/categorical label model is binary/multi-class cross-entropy. With Gaussian noise it's squared error.
7. MLE overfits with little data: one heads gives $\hat\phi = 1$, and an unseen class gets probability 0.