Neural networks
You already know one neuron — it's logistic regression. This chapter stacks neurons into a network, counts its parameters, trains a three-neuron network by hand with the chain rule, and builds the multivariate chain rule that everything deeper depends on.
The Neural Networks pre-reading has a few slips; this chapter uses the corrected versions and flags each where it comes up:
- A layer of $K$ neurons on $M$ inputs has $KM + K = K(M+1)$ parameters — the notes write "$KM + M$".
- The 3-3-2-1 example sums to 23 ($12 + 8 + 3$); the notes' line says "Total: 21", though their own summary table says 23.
- The three-neuron network's parameter list omits $w_{1,2}$ (it still correctly says 7 parameters).
- In the logistic-regression derivative table, the bias line simplifies to $-\frac1N\sum(1-\hat y)$, which is only the $y = 1$ case; the general answer is $-\frac1N\sum(y - \hat y)$.
- §2.2.3 ("Chain rule with multiple inputs") is cut off after its first paragraph.
1. Linear and logistic regression, redrawn as networks
Neural-network diagrams look intimidating, but the notes' first move is to redraw two models you already know in network notation. Inputs on the left, a $\Sigma$ node that computes a weighted sum plus bias, then (for logistic regression) a nonlinearity $g$, then the loss $\ell$ comparing the output $\hat y$ with the label $y$.
| Linear regression | Logistic regression | |
|---|---|---|
| Model | $\hat y = b + \sum_j w_j x_j$ | $\hat y = g_{\text{logistic}}\bigl(b + \sum_j w_j x_j\bigr)$ |
| Parameters | $w_1, w_2, b$ | |
| Loss | squared error $(y - \hat y)^2$ | cross-entropy |
| Objective | $J(\theta) = \frac1N\sum_{i=1}^N \ell(y^{(i)}, \hat y^{(i)})$ | |
| $\partial J/\partial w_j$ via chain rule | $\frac1N\sum_i \frac{\partial\ell}{\partial\hat y}\frac{\partial\hat y}{\partial w_j} = -\frac2N\sum_i (y^{(i)} - \hat y^{(i)})x_j^{(i)}$ | $\frac1N\sum_i \frac{\partial\ell}{\partial\hat y}\frac{\partial\hat y}{\partial z}\frac{\partial z}{\partial w_j} = -\frac1N\sum_i (y^{(i)} - \hat y^{(i)})x_j^{(i)}$ |
| $\partial J/\partial b$ | $-\frac2N\sum_i (y^{(i)} - \hat y^{(i)})$ | $-\frac1N\sum_i (y^{(i)} - \hat y^{(i)})$ |
| SGD update | $w_j \leftarrow w_j - \alpha\,\frac{\partial J^{(i)}}{\partial w_j}, \qquad b \leftarrow b - \alpha\,\frac{\partial J^{(i)}}{\partial b}$ | |
The point of the table is the derivative row. Each derivative is a product of local derivatives, one per box the signal passes through — $\frac{\partial\ell}{\partial\hat y}$, then $\frac{\partial\hat y}{\partial z}$, then $\frac{\partial z}{\partial w_j}$. The logistic middle factor is $\sigma'(z) = \hat y(1-\hat y)$, and it cancels against the cross-entropy's $\frac{1}{\hat y}$-type terms exactly as in chapter 11 §7.3. A neural network's gradient is the same idea with longer products.
2. The neuron and its activation function
A neuron is a linear function followed by an activation function:
$$ z = b + \sum_j w_j x_j, \qquad a = g(z). $$The activation $g$ is (almost) any nonlinear function. Logistic regression is a neuron with $g = \sigma$.
| Activation | $g(z)$ | $g'(z)$ | Notes |
|---|---|---|---|
| Logistic (sigmoid) | $\frac{1}{1+e^{-z}}$ | $\sigma(z)(1-\sigma(z))$ | Output in $(0,1)$; saturates at both ends |
| ReLU (rectified linear unit) | $\max(0, z)$ | $1$ if $z > 0$, else $0$ | "Takes the input and clamps negative values to zero." The modern default |
| Threshold (historic) | $\1\{z \ge 0\}$ | $0$ almost everywhere | The original perceptron; untrainable by gradient descent |
Stack two neurons with no activation: $w_2(w_1 x + b_1) + b_2 = (w_2w_1)x + (w_2b_1 + b_2)$ — still a line. Any stack of linear layers collapses into a single linear layer, so depth buys nothing. The nonlinearity is what lets a network represent curves.
3. A three-neuron network
The notes' example: predict traffic volume from time of day. The data rises through the morning and falls through the evening, so a line in $x_1$ alone won't do. You've seen one fix — engineer $x_2 = x_1^2$ and fit a quadratic with linear regression (chapter 10). Here's the other: two ReLU neurons, one to model the morning's upward trend and one the evening's downward trend, and a third that combines them.
$w_1, w_2, b$ — 3 parameters. You chose the feature $x_1^2$.
$w_{1,1}, b_{1,1}, w_{1,2}, b_{1,2}, w_{2,1}, w_{2,2}, b_{2,1}$ — 7 parameters. The network learns its own features $a_{1,1}, a_{1,2}$.
In the code that drives Figure 3, starts #1–#20 (the one shown on load plus the next 19 presses of "New random start"), each trained 3000 steps at $\alpha = 0.1$, ended with MSE ranging from $0.0018$ to $0.0315$; 8 of the 20 finished above $0.02$, most of them with a dead neuron. The good fits all put one hinge at about 9 am. The quadratic regression's MSE is $0.0090$ — and it is the same every time, because its objective is convex with a closed-form solution.
The derivatives, by the chain rule
Use squared error and one training point ($N = 1$), so $J = (y - \hat y)^2$ and $\frac{\partial J}{\partial\hat y} = -2(y - \hat y)$. Follow each parameter's path to the output and multiply the local derivatives along it:
| Parameter | Chain | Evaluates to |
|---|---|---|
| $b_{2,1}$ | $\frac{\partial J}{\partial\hat y}\frac{\partial\hat y}{\partial b_{2,1}}$ | $-2(y-\hat y)\cdot 1$ |
| $w_{2,j}$ | $\frac{\partial J}{\partial\hat y}\frac{\partial\hat y}{\partial w_{2,j}}$ | $-2(y-\hat y)\cdot a_{1,j}$ |
| $b_{1,j}$ | $\frac{\partial J}{\partial\hat y}\frac{\partial\hat y}{\partial a_{1,j}}\frac{\partial a_{1,j}}{\partial z_{1,j}}\frac{\partial z_{1,j}}{\partial b_{1,j}}$ | $-2(y-\hat y)\cdot w_{2,j}\cdot \1\{z_{1,j} > 0\}\cdot 1$ |
| $w_{1,j}$ | $\frac{\partial J}{\partial\hat y}\frac{\partial\hat y}{\partial a_{1,j}}\frac{\partial a_{1,j}}{\partial z_{1,j}}\frac{\partial z_{1,j}}{\partial w_{1,j}}$ | $-2(y-\hat y)\cdot w_{2,j}\cdot \1\{z_{1,j} > 0\}\cdot x_1$ |
Every update is then $\text{param} \leftarrow \text{param} - \alpha\,\frac{\partial J}{\partial\,\text{param}}$ — the same SGD rule as always, just applied to 7 parameters.
$\frac{\partial J}{\partial w_{1,j}}$ = (how wrong the output is) × (how much the output listens to neuron $j$) × (is neuron $j$ switched on?) × (how big the input was). If the neuron is off ($z_{1,j} \le 0$), the whole product is zero — that's the dead-ReLU effect from Figure 3, written as algebra.
Parameters: $w_{1,1} = 1,\ b_{1,1} = -1,\ w_{1,2} = -1,\ b_{1,2} = 3,\ w_{2,1} = 2,\ w_{2,2} = 1,\ b_{2,1} = 0.5$. One training point $x_1 = 2$, $y = 4$. Squared error. Compute the forward pass, the loss, all 7 gradients, and one SGD step with $\alpha = 0.02$. Did the loss go down?
Step 1 — Forward pass
$z_{1,1} = 1\cdot 2 - 1 = 1$, $a_{1,1} = 1$. $z_{1,2} = -1\cdot 2 + 3 = 1$, $a_{1,2} = 1$.
$\hat y = 0.5 + 2(1) + 1(1) = 3.5$. Loss $J = (4 - 3.5)^2 = 0.25$.
Step 2 — The shared factor
$\frac{\partial J}{\partial\hat y} = -2(4 - 3.5) = -1$. Every gradient starts with this. Both neurons are on ($z > 0$), so both ReLU factors are $1$.
Step 3 — All seven gradients
| param | gradient | new value ($\alpha = 0.02$) |
|---|---|---|
| $b_{2,1}$ | $-1$ | $0.52$ |
| $w_{2,1}$ | $-1\cdot a_{1,1} = -1$ | $2.02$ |
| $w_{2,2}$ | $-1\cdot a_{1,2} = -1$ | $1.02$ |
| $b_{1,1}$ | $-1\cdot w_{2,1} = -2$ | $-0.96$ |
| $w_{1,1}$ | $-1\cdot w_{2,1}\cdot x_1 = -4$ | $1.08$ |
| $b_{1,2}$ | $-1\cdot w_{2,2} = -1$ | $3.02$ |
| $w_{1,2}$ | $-1\cdot w_{2,2}\cdot x_1 = -2$ | $-0.96$ |
Step 4 — Check the new loss
$z_{1,1} = 1.08\cdot 2 - 0.96 = 1.2$; $z_{1,2} = -0.96\cdot 2 + 3.02 = 1.1$.
$\hat y = 0.52 + 2.02(1.2) + 1.02(1.1) = 0.52 + 2.424 + 1.122 = 4.066$. New loss $(4 - 4.066)^2 = 0.00436$.
Answer & the takeaway
Loss fell from $0.25$ to $0.00436$. Two lessons. (1) Layer-1 gradients are bigger here ($-4$ for $w_{1,1}$) because they're multiplied by the downstream weight and the input — early layers can be very sensitive. (2) The step size matters a lot: with $\alpha = 0.1$ the same gradients overshoot to $\hat y = 6.45$ and a loss of $6.0025$ — 24× worse than where you started. (3) Now suppose instead $x_1 = 4$: then $z_{1,2} = -1\cdot 4 + 3 = -1 < 0$, neuron 2 is off, and all of $w_{1,2}, b_{1,2}$'s gradients are zero for that point.
4. Fully connected networks
Once you can build three neurons, you can build any number. A fully connected network organizes neurons into $L$ layers, and each neuron in a layer takes every output of the previous layer as input (the first layer takes every input feature). If the previous layer has $M$ values, each neuron's linear function has $M$ inputs.
$w_{2,1,3}$ is the weight in layer 2, going into neuron 1 of that layer, from neuron 3 of the previous layer. The model for the notes' 3-3-2-1 network, layer by layer:
$$ a_{1,i} = g(z_{1,i}),\quad z_{1,i} = b_{1,i} + \sum_j w_{1,i,j}\,x_j $$ $$ a_{2,i} = g(z_{2,i}),\quad z_{2,i} = b_{2,i} + \sum_j w_{2,i,j}\,a_{1,j} $$ $$ \hat y = a_{3,1} = g(z_{3,1}),\quad z_{3,1} = b_{3,1} + \sum_j w_{3,1,j}\,a_{2,j} $$Substituted into one line: $\hat y = g\Bigl(b_{3,1} + \sum_k w_{3,1,k}\,g\bigl(b_{2,k} + \sum_i w_{2,k,i}\,g(b_{1,i} + \sum_j w_{1,i,j}x_j)\bigr)\Bigr)$. Nested functions — which is why the derivatives will be long chain rules.
Counting parameters
A neuron with $M$ inputs has $M$ weights and $1$ bias. A fully connected layer of $K$ neurons on $M$ inputs therefore has
$$ \underbrace{KM}_{\text{weights}} + \underbrace{K}_{\text{biases}} = K(M+1) \text{ parameters.} $$The notes write "$KM + M = K(M+1)$". The middle expression is a typo: there is one bias per neuron, so $K$ of them, not $M$. ($KM + M = M(K+1)$, which isn't equal to $K(M+1)$ unless $K = M$ — which is why the error is invisible in the notes' 3-input, 3-neuron first layer.)
3 inputs → 3 neurons → 2 neurons → 1 output neuron. How many parameters?
Layer by layer
Layer 1: $3\cdot 3 + 3 = 12$. Layer 2: $2\cdot 3 + 2 = 8$. Layer 3: $1\cdot 2 + 1 = 3$.
Answer & the takeaway
$12 + 8 + 3 = \mathbf{23}$ — matching the notes' table ($w_{1,i,j}$: 9, $b_{1,i}$: 3, $w_{2,i,j}$: 6, $b_{2,i}$: 2, $w_{3,1,j}$: 2, $b_{3,1}$: 1). The notes' "Total: 21" is an arithmetic slip. And the §3 network is 1-2-1: $2(1+1) + 1(2+1) = 7$. ✓
Vocabulary
| Output layer | The last layer of neurons |
| Hidden layers | All layers of neurons before it |
| "Input layer" | The input data itself — careful: it's not a layer of neurons, and has no parameters |
| Fully connected = linear = dense layer | Three names for the same thing |
| Multi-layer perceptron (MLP) | A fully connected network. Slightly misleading: a perceptron is a single neuron with a hard threshold activation, while modern networks use smooth-ish activations like ReLU or sigmoid |
5. Derivatives in a deeper network
The notes show just the bias derivatives for the 3-3-2-1 network ("less crazy indexing"):
$$ \frac{\partial J}{\partial b_{3,1}} = \frac{\partial\ell}{\partial\hat y}\frac{\partial a_{3,1}}{\partial z_{3,1}}\frac{\partial z_{3,1}}{\partial b_{3,1}} $$ $$ \frac{\partial J}{\partial b_{2,i}} = \frac{\partial\ell}{\partial\hat y}\frac{\partial a_{3,1}}{\partial z_{3,1}}\frac{\partial z_{3,1}}{\partial a_{2,i}}\frac{\partial a_{2,i}}{\partial z_{2,i}}\frac{\partial z_{2,i}}{\partial b_{2,i}} $$ $$ \frac{\partial J}{\partial b_{1,i}} = \frac{\partial\ell}{\partial\hat y}\frac{\partial a_{3,1}}{\partial z_{3,1}}\frac{\partial z_{3,1}}{\partial \mathbf a_{2}}\frac{\partial \mathbf a_{2}}{\partial \mathbf z_{2}}\frac{\partial \mathbf z_{2}}{\partial a_{1,i}}\frac{\partial a_{1,i}}{\partial z_{1,i}}\frac{\partial z_{1,i}}{\partial b_{1,i}} $$Look at the last line: the middle factors are in bold — vectors. Why? Because $b_{1,i}$ affects $a_{1,i}$, and $a_{1,i}$ feeds both layer-2 neurons. There are two paths from $b_{1,i}$ to $\hat y$. The single-variable chain rule, which multiplies along one path, isn't enough. The notes: "as we start to have cycles in our network diagram" (they mean paths that split and rejoin) "we'll need the multivariate chain rule." That's §6.
6. The chain rule on computation graphs
6.1 One path: multiply along the edges
For $h(x) = f(g(x))$, the chain rule is $\frac{d}{dx}f(g(x)) = f'(g(x))\,g'(x)$, or in Leibniz notation $\frac{df}{dx} = \frac{df}{dg}\frac{dg}{dx}$. Draw the computation as a graph $x \to g \to f$ and write each edge's local derivative on it: $\frac{dg}{dx}$ on the first edge, $\frac{df}{dg}$ on the second. The derivative of the output with respect to the input is the product of the edge derivatives along the path. It extends to any depth:
$$ h(x) = f_4(f_3(f_2(f_1(x)))) \quad\Longrightarrow\quad \frac{df_4}{dx} = \frac{df_4}{df_3}\frac{df_3}{df_2}\frac{df_2}{df_1}\frac{df_1}{dx}. $$6.2 Several paths: sum over them
Now $h(x) = f(g_1(x), g_2(x))$ with $g_1, g_2 : \R\to\R$ and $f : \R^2 \to \R$. $x$ feeds both $g$'s, so there are two paths from $x$ to the output. Each path's product is one way $x$ influences $h$; the total influence is their sum:
$$ \frac{dh}{dx} = \frac{\partial f}{\partial g_1}\frac{dg_1}{dx} + \frac{\partial f}{\partial g_2}\frac{dg_2}{dx}. $$For $f:\R^n\to\R$ and $g_1,\dots,g_n:\R\to\R$, $$ \frac{d}{dx}f\bigl(g_1(x),\dots,g_n(x)\bigr) = \sum_{i=1}^{n}\frac{\partial f}{\partial g_i}\frac{dg_i}{dx}. $$ Multiply along each path, then add over paths. The one-path rule is the $n = 1$ case.
$h(x) = f(g_1(x), g_2(x))$ with $g_1 = x^2$, $g_2 = \sin x$, $f(u, v) = uv$. Find $h'(1)$ using the multivariate chain rule.
Step 1 — Edge derivatives
$\frac{dg_1}{dx} = 2x = 2$. $\frac{dg_2}{dx} = \cos x = \cos 1 = 0.5403$. $\frac{\partial f}{\partial g_1} = g_2 = \sin 1 = 0.8415$. $\frac{\partial f}{\partial g_2} = g_1 = 1$.
Step 2 — Multiply along, add across
Answer & the takeaway
$2.2232$ — and the product rule on $x^2\sin x$ gives $2x\sin x + x^2\cos x = 2.2232$ too. The product rule is the multivariate chain rule for $f(u,v) = uv$. A finite difference with step $10^{-6}$ gives $2.2232$ as well.
6.3 Vector form
Stack the inner functions into one vector-valued function $\mathbf g(x) = [g_1(x), \dots, g_n(x)]^\top$, so $h(x) = f(\mathbf g(x))$ — the graph now has one path through a vector node. In denominator layout, $\frac{\partial\mathbf g}{\partial x} = \bigl[\frac{\partial g_1}{\partial x}\ \cdots\ \frac{\partial g_n}{\partial x}\bigr]$ (a row) and $\frac{\partial f}{\partial\mathbf g} = \bigl[\frac{\partial f}{\partial g_1}, \dots, \frac{\partial f}{\partial g_n}\bigr]^\top$ (a column). Multiplying along the single edge-path:
$$ \frac{d}{dx}f(\mathbf g(x)) = \frac{\partial\mathbf g}{\partial x}\,\frac{\partial f}{\partial\mathbf g} = \sum_{i=1}^n \frac{\partial f}{\partial g_i}\frac{dg_i}{dx}, $$— a dot product, which is exactly the sum over paths. Same computation, same derivative, but now written with vectors, which is how the bold $\partial\mathbf a_2/\partial\mathbf z_2$ factors in §5 are handled.
Once derivatives are vectors and matrices, order matters. In denominator layout the chain runs inner-to-outer, left to right ($\frac{\partial\mathbf g}{\partial x}\frac{\partial f}{\partial\mathbf g}$); numerator layout runs the other way. Pick one layout, stick with it, and check that the shapes of every product line up.
The notes' §2.2.3 ("chain rule with multiple inputs" — vector inputs and vector outputs) stops after its opening paragraph, which says only that the general case is repeated application of the rule above. Lecture 12 covers it — see chapter 14 for the vector version and the linear layer's gradients.
7. Recap
- A neuron = linear function + nonlinear activation. Logistic regression is one neuron.
- Activations: sigmoid, ReLU $\max(0,z)$; the historic threshold (perceptron) has zero gradient. Without nonlinearity, layers collapse to one linear map.
- Three-neuron traffic network: two ReLU hinges + a combining neuron, 7 parameters, vs 3 for linear regression with $x_1^2$. The network learns its features; its objective is not convex, so the result depends on initialization.
- Fully connected: every neuron sees every output of the previous layer. Notation $w_{\text{layer,output,input}}$.
- A layer of $K$ neurons on $M$ inputs has $K(M+1)$ parameters; 3-3-2-1 has $12 + 8 + 3 = 23$.
- Output layer / hidden layers; the "input layer" is data, not neurons. Fully connected = linear = dense; MLP.
- Gradients: multiply local derivatives along each path from parameter to loss; sum over paths when paths split (multivariate chain rule); in vector form, a dot product.