Part II · Chapter 15

Computer vision: convolutional networks and ResNets

Why a plain fully connected network is the wrong tool for images, what a convolution actually computes (by hand and interactively), how pooling and stacking build a CNN, what made AlexNet a turning point, and why adding layers made networks worse until skip connections fixed it.

1. Images, and why the models you know struggle

A grayscale image is a grid of pixels, each a number in $[0, 1]$ (0 = black, 1 = white). A colour image is three such grids — red, green, blue — called channels. Suppose you have labelled images and want to classify them. The lecture asks two questions and leaves the first open.

Logistic regression with pixels as features?

Flatten the grid into a long vector and fit $\sigma(\boldsymbol\theta^\top\mathbf x)$. It runs. But the model is one weight per pixel — effectively a single template image that it dot-products with the input. It can recognise "bright pixels in roughly these places", and nothing more: a cat shifted 5 pixels left lights up a different set of weights, and its decision boundary is a single hyperplane in pixel space. It works on very tidy datasets (centred digits); it fails on real photos.

A fully connected neural network?

Better — it can learn features. But the lecture lists three problems, and they're all one root cause: a fully connected layer treats the image as a bag of unrelated numbers.

Doesn't exploit spatial relations. Neighbouring pixels are strongly related (they're usually part of the same object). An FC layer has no notion of "neighbour". Here's a striking way to see it: shuffle every image's pixels with the same fixed permutation. An FC network learns exactly as well on the scrambled images — it just learns permuted weights. A human couldn't recognise anything. So the FC network isn't using the structure humans rely on.
Not spatially invariant. A pattern learned in the top-left corner has to be re-learned, from separate weights, for every other location.
Too many parameters. A 224×224 colour image has $224\cdot224\cdot3 = 150{,}528$ inputs. One hidden layer of 1,000 neurons needs $150{,}528\times1{,}000 + 1{,}000 = 150{,}529{,}000$ parameters — for one layer. That's a lot of compute and, worse, a lot of data needed to fit it without overfitting.

Figure 1 — the same pattern, shifted: a fixed template (FC-style) vs a sliding detector (conv-style)

Shift the pattern right

The image contains a 3×3 "plus" (5 bright pixels). Left: an FC-style detector — one weight per pixel, equal to the plus at its original position (outlined). Its score is the dot product with the whole image. Right: a conv-style detector — the same 3×3 plus as a filter, slid over every position, keeping the largest response. What to notice: shift by even one pixel and the fixed template's score drops; the sliding filter's score doesn't change at all, because it finds the pattern wherever it is.

Two families of image-specific model fix this:

Today

Convolutional Neural Networks (CNNs)

Less compute; work better with limited data. Bake in "nearby pixels matter" and "a pattern means the same thing anywhere".

Later in the course

Vision Transformers (ViTs)

Many more parameters; better with very large datasets. Transformers come later, in the context of language. Many modern models are hybrids of the two.

Image generation mostly uses a third paradigm, diffusion models — covered in 07-380, not here.

2. Convolution: a small window slid across the image

Start where the lecture does: you want to blur an image. Blurring means replacing each pixel with an average of itself and its neighbours. Do that by sliding a small grid of weights — a filter (also called a kernel) — across the image, and at each position multiplying the overlapping numbers and adding them up. That operation is convolution.

Analogy — a stencil scanned over a page

Hold a small stencil over the top-left of a page, score how well what's underneath matches it, write the score down, slide one step, repeat. The page of scores is the output — a feature map. A blur stencil scores "average brightness here"; an edge stencil scores "how sharp a light-dark change is here".

Try itThe lecture's blur example

Image $x = \begin{bmatrix}0.1 & 0.2 & 0.6\\ 0.3 & 0.4 & 0.9\\ 0.7 & 0.1 & 0\end{bmatrix}$, blur filter $\omega = \begin{bmatrix}0.25 & 0.25\\ 0.25 & 0.25\end{bmatrix}$. Compute the output.

Step 1 — What's being asked, and what size is the answer?

The 2×2 window fits in $3 - 2 + 1 = 2$ positions across and 2 down, so the output is 2×2. Each entry = sum of (window values × filter values) = the average of 4 pixels.

Step 2 — Top row of the output

Top-left window $\{0.1, 0.2, 0.3, 0.4\}$: $0.25(0.1+0.2+0.3+0.4) = 0.25$.
Top-right window $\{0.2, 0.6, 0.4, 0.9\}$: $0.25(2.1) = 0.525$.

Step 3 — Bottom row

Bottom-left $\{0.3, 0.4, 0.7, 0.1\}$: $0.3\times0.25 + 0.4\times0.25 + 0.7\times0.25 + 0.1\times0.25 = 0.375$ (the lecture's worked entry).
Bottom-right $\{0.4, 0.9, 0.1, 0\}$: $0.25(1.4) = 0.35$.

Answer & the takeaway

$z = \begin{bmatrix}0.25 & 0.525\\ 0.375 & 0.35\end{bmatrix}$. Two things to keep: the output is smaller than the input ($n - k + 1$ per side), and the same four weights were reused at every position.

Convolution formula (single channel)

For a $k_1\times k_2$ kernel $\omega$ and input $x$,

$$ z[i, j] = \sum_{u=0}^{k_1-1}\sum_{v=0}^{k_2-1} x[i+u,\ j+v]\cdot\omega_{u,v}. $$

$z[i,j]$ is the output at row $i$, column $j$. The double sum runs over the window whose top-left corner is at $(i, j)$: $u$ steps down, $v$ steps right. An $n_1\times n_2$ input gives an output of size $(n_1 - k_1 + 1)\times(n_2 - k_2 + 1)$.

going deeper In maths, "convolution" flips the kernel first; what's written above (no flip) is technically cross-correlation. Deep-learning libraries all compute this unflipped version and call it convolution. Since the kernel is learned, the flip makes no difference.

Figure 2 — convolution explorer: click pixels to paint, pick a filter, slide the window

Filter Window position

Input: 8×8, values 0 (dark) to 1 (bright) — click a cell to toggle it. Output cells are shaded blue for positive and pink for negative, darker = larger. What to notice: the vertical-edge filter $\begin{bmatrix}1&0&-1\end{bmatrix}$ (repeated in 3 rows) is zero over flat regions and lights up only where brightness changes left-to-right — positive on one side of the bright bar, negative on the other. Switch to the horizontal-edge filter and the bar's long sides vanish from the output; only its top and bottom ends light up, because those are the only places brightness changes top-to-bottom. A filter is a pattern detector, and what it detects is entirely set by its numbers.

3. Classical vision vs CNNs: who designs the filter?

Classical image processing

Many filters: blur, edge detection, …
Designed by hand
Convolved with the image to extract features
Features then fed to a downstream model

The CNN idea

Learn the filters from data. A 2×2 filter becomes four parameters

$$ \begin{bmatrix}\omega_{00} & \omega_{01}\\ \omega_{10} & \omega_{11}\end{bmatrix} $$

trained by gradient descent like any other weights.

This is chapter 12's "feature engineering vs feature learning" again. The edge detector in Figure 2 was hand-designed; a CNN would discover something like it (or something better that nobody thought of) because it reduces the loss.

Multi-channel convolution

A colour image has $C = 3$ channels; a grayscale one has $C = 1$. The filter gets a matching depth — it's a $C\times k_1\times k_2$ block — and the sum also runs over channels:

$$ z[i, j] = \sum_{u=0}^{k_1-1}\sum_{v=0}^{k_2-1}\sum_{c=0}^{C-1} x[c,\ i+u,\ j+v]\cdot\omega_{c,u,v}\ (+\ b). $$

Note the output is still one 2-D map: the filter mixes all the input channels together. A bias $b$ (one number per filter) is added too.

Try itCount the parameters

A convolutional layer takes a 224×224 RGB image and applies 64 filters, each 3×3, with a bias each. Stride 1, no padding. (a) How many parameters? (b) What's the output shape? (c) Compare with §1's fully connected layer.

Step 1 — One filter

Depth must match the input: $3\times3\times3 = 27$ weights, plus 1 bias = 28.

Step 2 — All filters, and the output

$64\times 28 = 1{,}792$ parameters. Each filter produces a $(224-3+1)\times(224-3+1) = 222\times222$ map, so the output is $64\times222\times222$ — one map per filter.

Answer & the takeaway

1,792 parameters vs 150,529,000 for the FC layer — about 84,000× fewer. The count doesn't depend on the image size at all, because the same small filter is reused at every position. That's parameter sharing, and it's where most of the saving comes from. Common mistake: forgetting the channel depth (answering $64\times(9+1) = 640$).

4. Many filters, pooling, and the CNN pipeline

One filter detects one kind of pattern. A layer runs several convolutions in parallel, each with its own kernel and bias, producing several feature maps. The next conv layer's filters then have depth equal to that number of maps — so it combines "edge here" and "edge there" into "corner".

The catch: many feature maps, each almost image-sized, is a lot of memory and compute. So we downsample each map — pooling. Split the map into small blocks (typically 2×2) and replace each block with one number:

Max pooling — keep the largest. "Did this feature appear anywhere in this block?"
Average pooling — keep the mean. "How strongly, on average?"

Pooling is applied to each feature map separately and has no parameters.

Figure 3 — 2×2 pooling on a 4×4 feature map

What to notice: each colour block becomes one number, so width and height halve (16 numbers → 4). Max pooling keeps the strongest response in the block, which also means a feature that moves by one pixel within a block gives the same output — a little built-in shift tolerance.

The typical CNN pipeline

Conv → Pool → Nonlinearity (ReLU) → Conv → Pool → ReLU → ⋯ → Fully Connected → Softmax

The conv/pool stages shrink the image spatially while growing the number of feature maps; the final FC layer + softmax is just multiclass logistic regression on the learned features (chapter 11).

Figure 4 — shapes through a small CNN (28×28 grayscale, 10 classes)

Two conv layers (5×5 filters, 6 then 16 of them), each followed by 2×2 max pooling and ReLU, then one FC layer to 10 classes. Parameter counts are written under each layer (weights + biases). What to notice: the second conv layer's filters are $6\times5\times5$ deep, not $5\times5$ — they read all 6 maps below them. The whole network has 5,142 parameters, and half of them are in the final FC layer.

An input is $32\times32\times3$. A conv layer has 10 filters of size $5\times5$ (stride 1, no padding), followed by 2×2 max pooling. What's the output shape?

Why: conv: $32 - 5 + 1 = 28$, one map per filter → $28\times28\times10$ (that's option A — correct but before pooling). Pool halves width and height → $14\times14\times10$. The input's 3 channels are summed inside each filter, so they don't multiply the number of maps (option B's mistake). Option D pooled the input instead of the conv output.

5. Properties of convolutional layers

Still linear in the parameters. Like an FC layer, the weights and bias apply linearly to the previous layer's outputs. A conv layer is an FC layer with most weights forced to zero and the rest forced to be copies of each other.
Parameter sharing. The same weights are used at every sliding-window location.
Sparse (local) connectivity. Each output node connects to only a small window of the previous layer, not to every node.
Trained by backprop. Nothing new is needed — with one detail from chapter 14 (below).
Hierarchical. Each layer combines the previous layer's features, and CNNs are empirically observed to learn interesting representations on their own.

Backprop through a shared weight = the "+=" rule

A filter weight $\omega_{u,v}$ is used at every output position $(i, j)$. That's fan-out, so its gradient is a sum over all of them:

$$ \frac{\partial J}{\partial\omega_{u,v}} = \sum_{i}\sum_{j}\frac{\partial J}{\partial z[i,j]}\; x[i+u,\ j+v]. $$

This is chapter 14's multivariate chain rule and its "+=" accumulator, applied to a weight used hundreds of times. (Fun fact: the right-hand side is itself a convolution — of the input with the output gradients.)

Figure 5 — what the layers learn (schematic reconstruction of the lecture's Figure 1)

The lecture's figure (credit Honglak Lee, NIPS 2010 workshop) shows real learned features as images: 1st layer ≈ oriented edges, 2nd ≈ object parts, 3rd ≈ whole objects (faces, in that example). This is a hand-drawn schematic of the same idea, not the learned features themselves — check the original slide for those. What to notice: each layer sees a bigger patch of the image than the one before (its receptive field grows), so it can represent bigger structures built from smaller ones.

6. AlexNet: the moment deep learning took over vision

ImageNet was a landmark competition because of its (then) sheer size and difficulty. Before it, there was far less emphasis on collecting gigantic datasets. In 2012, AlexNet — a CNN — got about 15% error. The next best entry got about 26%. That gap was so large that computer vision switched to deep learning almost overnight. Its key contributions:

ReLU activations

$\mathrm{ReLU}(x) = \max\{0, x\}$. Previous work used tanh or sigmoid, which are more expensive to compute and whose gradients aren't well behaved: the sigmoid's derivative is at most $0.25$ and near zero for large $|z|$, so stacking many layers shrinks the gradient (chapter 14, Figure 4). ReLU's derivative is exactly 1 wherever it's active.

Training on GPUs

Convolutions are millions of independent multiply-adds — exactly what graphics hardware does in parallel. This made a big network trainable in days instead of months.

Deeper architectures

More layers → more levels of the feature hierarchy in §5.

Data augmentation

Artificially enlarge the training set: from each 256×256 image, take several 224×224 patches. A 224 window fits in $256 - 224 + 1 = 33$ positions per side, so $33^2 = 1{,}089$ possible crops — each still clearly the same object, so each a valid new labelled example. (The AlexNet paper also used horizontal flips.)

Dropout

To fight overfitting, at each training gradient step randomly drop a subset of the network's connections (the lecture says "edges"; the standard implementation zeroes a random subset of neurons' outputs, which drops all their outgoing edges at once). At test time nothing is dropped.

Intuition: no neuron can rely on any particular other neuron being there, so the network can't build fragile, co-adapted pathways that memorise individual training examples. It's a bit like training a large team where random members are absent each day — everyone has to be broadly useful.

Figure 6 — why ReLU: the derivative of each activation

What to notice: the sigmoid's derivative (blue) never exceeds 0.25 and is almost 0 beyond $|z| \approx 5$. Backprop multiplies one of these per layer, so 10 sigmoid layers scale the gradient by at most $0.25^{10} \approx 0.001$. ReLU's derivative (pink) is 1 on the whole active side, so it passes gradients through undiminished.

7. ResNets: when deeper got worse

Residual networks were invented for CNNs but are now used almost everywhere (including transformers). Start with the puzzle that motivated them.

Think about itThe lecture's question: 20 layers or 56?

Take a 20-layer network and a 56-layer network. The 20-layer one is a special case of the 56-layer one: for any parameters $(\omega, b)$ of the 20-layer net there are parameters $(\omega', b')$ of the 56-layer net with $h_{\omega,b}(x) = h'_{\omega',b'}(x)$ for every input $x$. Train both with ERM on the same data. Which has lower training error?

Step 1 — Why is the 20-layer net a special case?

Copy its 20 layers into the first 20 of the 56, and make the remaining 36 layers each compute the identity (output = input). Then the 56-layer net computes exactly the same function.

Step 2 — The expected answer

The 56-layer net can do everything the 20-layer net can, and more. ERM minimises training error over everything the model can represent, so its best training error can only be lower or equal. Expect: 56 layers ≤ 20 layers on training error.

Step 3 — What was actually observed

The 56-layer network had higher training error — and higher test error too. (The lecture's plots show training-error and test-error curves over iterations, with the 56-layer curve above the 20-layer curve in both.)

The takeaway — it's not overfitting

Overfitting means low training error and high test error. Here training error is higher — so the bigger model isn't fitting too well, it's failing to fit at all. The problem isn't what the model can represent; it's that gradient descent can't find the good parameters. It's an optimisation problem.

What's going on: vanishing (and exploding) gradients

Picture the network as a chain: input → layer 1 → layer 2 → ⋯ → layer $L$ → output. By backprop (chapter 14), the gradient reaching layer 1 is a product of one local derivative per layer to its right. If those factors are typically smaller than 1, the product shrinks exponentially with depth (vanishes); if larger than 1, it grows exponentially (explodes). Either way, the weights on the left either hardly change during training or change chaotically.

Why "learning zero is easier than learning identity"

The lecture's key observation. To make an extra layer harmless, a plain network has to learn the identity map exactly — weights precisely tuned so that output = input. But weights are initialised small and random, so a fresh layer computes roughly zero, not the identity, and pushing it to exactly identity is a delicate target. The fix: change what "do nothing" means, so that all-zero weights are the identity.

Skip (residual) connection

Add a block's input directly to its output: instead of $\mathbf h_{\text{out}} = F(\mathbf h_{\text{in}})$, compute

$$ \mathbf h_{\text{out}} = F(\mathbf h_{\text{in}}) + \mathbf h_{\text{in}}. $$

$F$ only has to learn the residual — the change to make. If the block should do nothing, $F = 0$: all-zero weights, the easy target.

Figure 7 — a skip connection (the lecture's diagram)

What to notice: there are now two routes from layer 1's output to the rest of the network — through layer 2, and around it. In the backward pass the gradient takes both and they add (the multivariate chain rule again), so the skip route delivers the gradient back with a factor of exactly 1, no matter what layer 2 does.

In derivative terms: for one block, $\frac{\partial\mathbf h_{\text{out}}}{\partial\mathbf h_{\text{in}}} = \frac{\partial F}{\partial\mathbf h_{\text{in}}} + I$. The "+ I" means each layer's factor is near 1 instead of near 0 when $F$'s weights are small, so the product over many layers no longer collapses.

Figure 8 — gradient size at each layer: plain chain vs residual chain (toy model)

Depth $L$ Weight size $w$

Toy model, not a real ResNet: every layer is a scalar linear map with weight $\pm w$ (signs fixed at random). Plain layer: $h \leftarrow \pm w\,h$, so its local derivative is $\pm w$. Residual layer: $h \leftarrow h \pm w\,h$, local derivative $1 \pm w$. Bars show $\log_{10}$ of the gradient magnitude reaching each layer from the output (0 means size 1; −10 means $10^{-10}$). What to notice: the plain chain only survives in a knife-edge band near $w = 1$ — at $L = 56$, $w = 0.9$ shrinks layer 1's gradient to $0.9^{55} \approx 0.003$, and $w = 1.1$ blows it up to about 189. With small weights — which is how networks start out — the plain chain's gradient is gone: at $w = 0.1$, $L = 56$ it is $10^{-55}$ at layer 1, while the residual chain's is about 1.02 and stays between 0.79 and 1.55 at every layer. Now push $w$ up: the residual chain degrades too (about 0.19 at layer 1 for $w = 0.3$, about 0.002 for $w = 0.5$). Skip connections work best when each block makes a small change, and real ResNets add other tricks (careful initialisation, normalisation layers) to keep it that way.

With skip connections, the deeper network's training and test error both came out lower than the shallower one's — depth helped again, as the "special case" argument said it should.

A 56-layer plain network has higher training error than a 20-layer one. Which fix addresses the actual cause?

Why: high training error means the problem is optimisation, not overfitting — gradients vanish before reaching early layers. Dropout, augmentation and L2 all fight overfitting and would make training error worse if anything. Option D accepts the symptom; skip connections remove the cause and let much deeper nets train well.

8. Recap

Pixels as features ignore spatial structure; FC networks are not shift-invariant and need huge numbers of parameters ($150{,}529{,}000$ for one 1,000-unit layer on 224×224×3).
Convolution: slide a $k_1\times k_2$ filter, multiply-and-sum; $z[i,j] = \sum_u\sum_v x[i+u, j+v]\,\omega_{u,v}$; output $(n-k+1)$ per side. With channels, also sum over $c$.
CNNs learn the filters. Several filters per layer → several feature maps. Parameters per layer = filters × ($C k_1 k_2$ + 1), independent of image size.
Pooling (max or average, per map) downsamples. Pipeline: Conv → Pool → ReLU → ⋯ → FC → Softmax.
Conv layers: linear, shared weights, local connections, trained by backprop (shared weight ⇒ gradients summed over positions); layers learn edges → parts → objects.
AlexNet (2012, ~15% vs ~26%): ReLU, GPUs, depth, data augmentation (crops), dropout.
ResNets: deeper plain nets had higher training error — an optimisation failure from vanishing/exploding gradients. Skip connections $\mathbf h + F(\mathbf h)$ make "do nothing" = zero weights and pass gradients back with a factor of 1.