Part V · NeuroAI

NeuroAI: encoding, decoding & CNNs

Guest lecture, Dr. Maggie Henderson (CMU Psychology, Neuroscience Institute & Machine Learning Department). Artificial neural networks were built from a sketch of the visual system; now they are used as instruments to study it. Everything here is predicting one of two directions — stimulus to brain, or brain to stimulus — and the whole lecture turns on not confusing them.

Essentials

Encoding model: stimulus / task property → neural data. You feed in the image, you predict the brain response. It is an explicit hypothesis of the form "X is related to the brain via function F".
Decoding model: neural data → stimulus / task property. You feed in the brain response, you recover what the person saw, heard, remembered or intended. A decoder can be built by inverting an encoding model, or trained directly as a classifier.
NeuroAI: the intersection of neuroscience, AI and cognitive psychology. Two-way street — the brain inspired the architectures, and the architectures are now tools for investigating the brain.
BOLD fMRI: blood oxygen level dependent signal. A hemodynamic measure, using blood flow as a proxy for neural activity, because oxygenated and deoxygenated haemoglobin have different magnetic properties. Invented at Bell Labs in the 1990s; 3T and 7T scanners are common.
Voxel: a volume pixel — one small cube of tissue (commonly about 2 mm on a side) that gives one number per measurement. It pools hundreds of thousands of neurons, so a predicted voxel response is not a predicted neuron.
ANN / DNN / CNN: units ("neurons") connected by weights ("synapses"), weights learned from data during training; deep = many layers; convolutional = built from spatial filters. Inputs: images, sentences, game states. Outputs: captions, labels, sentences, actions.
Spatial convolution: slide a small 2D filter (a kernel) across the image, taking a dotwise product at every position. One set of weights, reused everywhere — so a feature is detected wherever it appears.
Ventral stream: retina → LGN → V1 (orientation, spatial frequency) → V2/V3/V4 (texture, contour, shape, colour) → IT (objects, scenes). Along it, feature complexity and receptive field size both increase. Early model layers best predict early visual areas; deeper layers best predict higher ones.
Brain alignment and BrainScore: a benchmark for standardised comparison of how well models predict neural data. Crucial caveat: high task performance does not guarantee high brain-prediction performance — which hints that object categorisation is not all vision evolved for.
Supervised / self-supervised / language-supervised: labels, structure in the data itself, paired captions. Language supervision helps most in higher visual areas, nearer semantic cortex.
Stimulus optimization: synthesise the image that maximises a region's predicted response (BrainDiVE, using a diffusion model that generates images by iteratively removing noise) instead of hand-picking photographs. Data-driven, so it can find preferences a predefined category list would miss — e.g. food-selective cortex.

1. What NeuroAI is, and why it goes both ways

Start with the problem. You want to know what features the brain uses to recognise objects and scenes. The classical move — the one the rest of this course has been doing — is to pick a hypothesis, build the stimuli that test it, and compare conditions: faces versus objects, intact versus scrambled, upright versus inverted. That works, and it produced the FFA, the LOC and the PPA. But it has a ceiling. You can only test features you already thought of, and you can only present a few dozen hand-chosen images.

Meanwhile, computer vision got good. Deep networks now map images to labels well enough to be useful, and they do it with representations nobody designed by hand. So there is a tempting trade: use those learned representations as candidate hypotheses about the brain's features, and let the data say which ones fit.

NeuroAI The rapidly growing field at the intersection of neuroscience, artificial intelligence and cognitive psychology. Dr. Henderson's toolkit is a good picture of it: experimental methods (human fMRI, human behaviour) plus computational methods (encoding and decoding models, neural network simulations).

The two directions of influence are worth separating, because the study guide asks for both.

Neuroscience → AI Deep convolutional networks were directly inspired by the architecture of the primate visual system: a hierarchy of stages, local receptive fields, simple features composed into complex ones. The word "neuron" in a network diagram is a historical debt, not a coincidence. Reinforcement learning's reward signals and recurrent networks' memory have similar lineages.

AI → neuroscience Today the traffic runs the other way, and that is the lecture's real subject. A trained network is a measuring instrument: its layers give you quantitative feature spaces to predict brain responses with (encoding), to read mental content out with (decoding), and to synthesise new stimuli with (stimulus optimization).

This connects to the framing from Marr's levels. An encoding model is a claim at the algorithmic level — these are the features and this is the transformation — tested against implementation-level data. It is the same logic as asking whether V1 simple cells compute oriented edges, scaled up to features too complicated to draw.

2. fMRI, BOLD and what a voxel actually is

Everything downstream depends on the measurement, so be precise about it. Chapter 8 covers fMRI as one method among many; here is the part the modelling rests on.

fMRI was invented at Bell Labs in the 1990s. A very strong magnetic field — 3T or 7T is typical — is used to detect tissue whose magnetic properties differ. The property that matters is the difference between oxygenated and deoxygenated haemoglobin. When a patch of cortex becomes active, local blood flow increases and oversupplies it with oxygen-rich blood, changing the ratio and therefore the signal. Hence BOLD: blood oxygen level dependent.

The one-sentence version BOLD fMRI is a hemodynamic measure: it uses blood flow as a proxy for neural activity. You are not watching neurons. You are watching the plumbing that feeds them, several seconds late.

A voxel is a "volume pixel": the scan divides the brain into a 3D grid of small cubes, and each cube returns one number per measurement. The BrainDiVE study in this lecture used 2 mm isotropic voxels with a 2 s repetition time, so each number summarises an 8 mm³ cube of cortex averaged over a couple of seconds. A whole-brain scan is tens of thousands of such numbers, which is precisely why these datasets can train models at all — the Natural Scenes Dataset (Allen et al., 2022) pairs tens of thousands of image presentations with voxel responses in a handful of people.

3. Why predicting a voxel is not predicting a neuron

This is asked directly on the study guide and it is the single easiest place to overclaim, so it gets its own section.

An 8 mm³ cube of cortex contains on the order of hundreds of thousands of neurons (at commonly quoted cortical densities of roughly 50,000–100,000 neurons per mm³, that is about 0.4–0.8 million per voxel; treat the figure as an order of magnitude, not a measurement). Every one of them, plus the glia, plus the blood vessels threading through, contributes to one number. Three consequences follow.

It is an average, and averages cancel. If half the neurons in a voxel prefer vertical and half prefer horizontal, the voxel looks untuned for orientation even though every neuron in it is sharply tuned.
It is indirect and slow. The signal is a vascular response that lags the spiking by seconds, so fine temporal structure — spike timing, the order in which two populations respond — is simply gone.
A good prediction constrains populations, not cells. "My model explains 60 percent of the variance in this voxel" means the model's features track what this chunk of cortex does on average. It does not mean any neuron computes that feature, and two very different neural codes can produce the same voxel pattern.

Common mistake Saying "the encoding model predicts neural activity" and sliding from there to "the model is doing what the neurons do". The honest claim is population-level: the model's feature space is aligned with the information this region's population carries. Single-neuron claims need single-neuron methods — electrophysiology, or the sEEG depth electrodes from Chapter 8.

Try itThe voxel that hides its own tuning

A voxel in V1 contains two equally sized neural populations. Population A fires at 10 spikes per second to a vertical grating and 2 to a horizontal one. Population B is the mirror image: 2 to vertical, 10 to horizontal. Assume the BOLD response of the voxel is proportional to the total spiking of all its neurons. What does an experimenter measuring this voxel conclude, and what is actually true?

Step 1 — What are we actually being asked?

Not "what do the neurons prefer" — we know that. We are asked what the measurement shows, and then what inference the measurement licenses. The question is about the gap between a pooled signal and its constituents.

Step 2 — Do the arithmetic

Vertical grating: population A contributes 10, population B contributes 2, total 12. Horizontal grating: A gives 2, B gives 10, total 12 again. The voxel responds identically to the two orientations.

Step 3 — So what does the experimenter conclude?

That this voxel is not orientation selective, and that an orientation-based encoding model should fail here. Both neural populations are in fact maximally orientation selective. The selectivity was destroyed by spatial averaging, not absent from the tissue.

Note the asymmetry this creates: a model that predicts a voxel well has found something real, but a model that fails on a voxel has not shown the information is missing from the cortex — it may only be invisible at 2 mm.

Answer & the takeaway

Measured: no orientation preference. True: every neuron is strongly tuned. The problem was built around the inference "voxel-level prediction equals neuron-level prediction". A null result in fMRI is weak evidence for absence, because the measurement averages over the very heterogeneity you care about. (This is also why ocular dominance and orientation columns, which alternate on a sub-millimetre scale, are so hard to see with standard fMRI — see Chapter 10.)

4. Encoding versus decoding: the direction is everything

Two models, built from the same data, pointing opposite ways. Students lose marks here for one reason only: they remember both words and not which arrow goes with which. So fix the arrow first and the definitions follow.

The two directions of prediction

What to notice: the two rows use the same three ingredients and differ only in which end is the input. The orange row (encoding) starts at the stimulus and ends at the brain; the teal row (decoding) starts at the brain and ends at a statement about the stimulus. Click a button to isolate one.

Encoding model Stimulus / task property → neural data.
Input: the image, or some property of it. Output: the predicted response of each voxel. It states an explicit hypothesis — "X is related to the brain via function F" — and is scored by how much response variance it explains on held-out stimuli.

Decoding model Neural data → stimulus / task property.
Input: the measured brain response. Output: which of 60 nouns the person read, which orientation they are holding in mind, what sentence they heard, which letter they intended to write. Either invert an encoding model, or train a generic classifier directly on the brain data.

How the encoding model is actually built

Mechanically it is a regression. Let $y$ be the measured response of one voxel across many images, and let $X$ be a matrix whose rows are images and whose columns are features extracted from each image — the activations of one layer of a CNN, say, or the outputs of a texture model. Fit

$$y = X\beta + \epsilon$$

where $\beta$ is the vector of weights (one per feature) telling you how much this voxel cares about each feature, and $\epsilon$ is noise. Because there are usually far more features than images, the fit uses ridge regression (L2 regularization), which shrinks the weights in $\beta$ toward zero and stops the model memorising the training set.

Then the scoring. Run the model on new images it never saw, compare predicted to actual, and compute the variance explained:

$$R^2 = 1 - \frac{SSE}{SST} = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2}$$

Here $y_i$ is the actual response on image $i$, $\hat{y}_i$ (the "hat" means predicted, not a power) is the model's prediction, and $\bar{y}$ is the mean response. The denominator is the variance you would fail to explain by always guessing the mean. Fitting one of these per voxel and painting the resulting $R^2$ values onto the cortical surface is how these papers' main figures are made — Henderson, Tarr & Wehbe (2023) did exactly this with a computational texture model.

Always on held-out data Compute $R^2$ on validation or test images only. A model with enough features can fit the training responses arbitrarily well — including the noise — and a training-set $R^2$ therefore measures nothing about the brain.

Linearizing encoding models

Why is the final step a plain linear regression, when the whole point was that visual features are complicated? Because the complexity is deliberately placed before the regression. The pipeline is: image → (very nonlinear feature mapping, e.g. a deep network) → features → (very simple linear map) → voxel response. The nonlinear part is the hypothesis; the linear part is just the readout.

Analogy Think of the feature model as a language and the linear weights as a translation dictionary. Keeping the dictionary word-for-word simple means that if the translation works, the credit belongs to the language, not to clever translating. That is what makes it a hypothesis test: "how well does my nonlinear feature mapping capture the neural encoding process?" (Naselaris et al., 2012). If you allowed a flexible nonlinear readout as well, a good fit would no longer tell you whose features were right.

Try itEncoding or decoding? Four studies

For each, name the model type and write the arrow with the input on the left.

Mitchell et al. (2008) scan people reading each of 60 nouns, then build a system that takes a new brain image and reports which noun the person was reading.
Huth et al. (2016) take word co-occurrence statistics for every word in a story, and use them to predict the response of each voxel while people listen to the story.
Willett et al. (2021) record from microelectrode arrays in the precentral gyrus of a paralysed patient and output the letter he is attempting to handwrite.
A lab feeds photographs through layer 4 of AlexNet and fits ridge regression from those activations to V4 voxel responses, reporting $R^2$ on held-out photographs.

Step 1 — What are we actually being asked?

Not "which is better" and not "what technique did they use". Only: what goes in and what comes out. Ignore how fancy the method is — a deep network can appear on either side. Ask what the system is handed at test time.

Step 2 — Work through them

Decoding. Neural data → which of 60 nouns. The input at test time is a brain image.
Encoding. Stimulus property (word co-occurrence statistics) → neural data. Note it used no language model at all; the features were plain corpus statistics.
Decoding. Neural data → intended letter. This is the BCI case from section 11.
Encoding. Stimulus → neural data, scored with $R^2$. The presence of a deep network is irrelevant to the direction.

Step 3 — The trap in number 2

Huth et al. is the one people mislabel, because the result is usually described as a map of "what each bit of cortex means" — which sounds like reading out from the brain. But the model was fit stimulus → brain. The semantic map is the pattern of fitted weights across voxels, with colours showing voxels with similar word tuning; it showed that semantic information is topographically organised rather than random. Output of analysis is not the same thing as output of the model.

Answer & the takeaway

Decoding, encoding, decoding, encoding. The problem was built around two confusions: that the sophisticated method tells you the direction (it does not), and that what the paper's figure shows tells you the direction (it does not either). Only the test-time input does. And remember the link between them: decoding can be done using an encoding model — invert it by asking which stimulus would have produced the observed response — or done generically with a classifier that never models the stimulus at all.

A model takes the orientation of a grating and predicts the BOLD response of every voxel in V1. This is…

Why: Direction is defined by what you feed in at test time: stimulus property → neural data is encoding. Option A is the classic reversal. Option C is tempting because the machinery is a regression — but "encoding model" names the direction, not the algorithm. Option D confuses what you could build later with what this model is: an encoding model can be inverted to make a decoder, but until you invert it, it only predicts brain data.

5. CNNs and the visual system: what they share

An artificial neural network (ANN) is a class of machine learning system inspired by the brain. Two pieces of vocabulary, and the brain word each one borrows:

In the network	Brain word it borrows	What it does
Units / nodes	"neurons"	Each holds one activation value, computed from the units feeding into it
Weights / edges	"synapses"	Scale the signal passing between units. The weights encode the information, and they are learned from data during training — not designed
Layers	cortical stages	A deep neural network (DNN) is simply an ANN with many layers

The same skeleton accepts very different jobs, which is worth knowing because the exam may ask for examples of either side:

Example inputs Images · sentences · game states

Example outputs Captions · labels · sentences · actions

So which principles are genuinely shared with the visual system? Four, and they are the answer to the study guide's question:

Hierarchy. Many stages in sequence, each taking the previous stage's output as input — matching retina → LGN → V1 → V2/V3/V4 → IT.
Local receptive fields. A unit early in the network sees only a small patch of the image, exactly as a V1 cell sees only a small patch of the visual field.
Growing complexity and growing receptive fields. Composing simple detectors into complex ones, with each stage pooling over a larger region.
Learned connection strengths. Information lives in the weights, which are shaped by experience, rather than in hand-coded rules.

Where the analogy leaks A unit is not a neuron: it has no spikes, no refractory period, no neurotransmitters, and typically no recurrence or feedback, whereas real visual cortex is massively recurrent and has top-down projections at every level. Training by backpropagation over millions of labelled images is not how a child learns to see. And the sharpest warning sign comes from the data itself: vision transformers, which were not brain-inspired at all, also predict brain responses well (Conwell et al., 2024). Architectural resemblance is clearly not necessary for good prediction — so a model predicting the brain well cannot be taken as proof that the brain works that way.

6. Spatial convolution, on an actual pixel grid

A digital image is a matrix of numbers (pixels). A filter — also called a kernel — is a smaller matrix of numbers. Spatial convolution takes the 2D filter and multiplies it, elementwise, against each part of the image, summing the products: a dotwise product at every position. You "slide" the filter across the whole image, and the resulting grid of sums is the feature map.

What is the sum measuring? Agreement. It is large and positive where the image pattern under the window matches the filter pattern, large and negative where it matches the opposite pattern, and near zero where there is nothing the filter is looking for. Convolution is a search for places where the image pattern matches the filter pattern.

Slide the kernel; watch the feature map fill in

Kernel position

What to notice: the image holds two one-pixel-wide vertical bars, one in the top left and one lower right. With the vertical-edge kernel, the same nine weights produce the same signature at both bars — plus 27 where the bar enters the window on the right, minus 27 where it leaves — even though the bars are four columns and four rows apart. Switch to the horizontal-edge kernel and those strong responses vanish: the largest magnitude anywhere drops to 9, and only at the tips where the bars start and stop. That is one filter, one feature, detected anywhere, and ignored when it is not there.

Weight sharing is the whole trick The filter's nine numbers do not change as it slides. So a feature learned once is detected everywhere in the image — the network does not need a separate vertical-edge detector for every location. This gives two things at once: far fewer parameters to learn, and translation invariance, meaning a shifted object produces a shifted feature map rather than an unrecognisable one. It is also where the resemblance to visual cortex is strongest: a V1-like filter repeated across a retinotopic map is the biological version of the same idea.

Try itOne convolution by hand

Here is the top-left 3 by 3 patch of the image in the figure above (rows 0–2, columns 0–2), and two kernels. Compute the output for this one window with each kernel.

Image patch	Vertical-edge kernel	Horizontal-edge kernel
0 0 9 0 0 9 0 0 9	−1 0 +1 −1 0 +1 −1 0 +1	−1 −1 −1 0 0 0 +1 +1 +1

Step 1 — What are we actually being asked?

For one fixed window position, multiply each of the nine image numbers by the kernel number in the same position, then add all nine products. One number out per kernel. No sliding yet, and no averaging — convolution is a sum of products, not a mean.

Step 2 — Vertical-edge kernel

Row by row, each row of the patch is (0, 0, 9) against (−1, 0, +1): 0·(−1) + 0·0 + 9·(+1) = 9. Three identical rows, so the total is 9 + 9 + 9 = 27.

Why large? The patch is dark on the left, bright on the right, and so is the kernel — maximum agreement.

Step 3 — Horizontal-edge kernel

Top row contributes 0·(−1) + 0·(−1) + 9·(−1) = −9. Middle row is multiplied by zeros, contributing 0. Bottom row contributes 0·(+1) + 0·(+1) + 9·(+1) = +9. Total: −9 + 0 + 9 = 0.

Why zero? The patch has no top-to-bottom change at all — every row is identical — so a filter asking "is the bottom brighter than the top?" gets exactly nothing. The bright pixels on the right cancel between the +1 and −1 rows.

Answer & the takeaway

27 and 0. Same pixels, same arithmetic, completely different answers — the filter, not the image, decides what counts as a feature. And because those nine weights are reused at every position, the same 27 shows up at the second bar further down the image, which you can confirm with the slider. The problem was built around two mistakes: thinking convolution averages the patch (it does not; it is a weighted sum that can be negative), and thinking a different location needs a different detector (it does not; that is weight sharing).

7. Along the ventral stream, and into the layers

The flow of information, which you already have from Chapter 9: eye (retina) → thalamus (LGN) → primary visual cortex (V1) → intermediate visual areas (V2, V3, V4) → higher visual cortex (IT, inferotemporal). What changes as you travel along it is the kind of thing represented:

Stage	Selective for	Receptive field
V1	Orientation, spatial frequency	Small
V4	Texture, contour, shape, colour	Medium
IT cortex	Objects, scenes	Large

Two things increase together along the hierarchy: the complexity of the features and the size of the receptive fields. Those two trends are not independent — you can only represent a whole object if you are looking at a region big enough to contain one. The payoff is a representation that is increasingly tolerant: an IT-level "this is a cat" survives a shift, a rescaling or a change in lighting that would completely change the V1 response.

Now the result that ties the two systems together. If CNNs really are a reasonable model of this pathway, then the network's depth ought to correspond to the brain's depth. Fit a separate encoding model from every layer to every visual area and see which layer wins where (Güçlü & van Gerven, 2015, using AlexNet):

Layer-to-area correspondence — click a model layer

What to notice: the linking lines do not cross. Click any layer to see the area it best predicts and what it is computing. The gradient is the claim: early model layers better predict early visual areas, deeper model layers better predict higher visual areas. This is a schematic of the correspondence, not plotted data — the original figure reports the best-predicting layer per area. reconstruction

Why this result matters so much Any single good prediction could be a coincidence of rich features. But an ordered correspondence — layer 1 for V1, layer 5 for IT, nothing crossing over — is a much harder pattern to get by accident. It helps validate the correspondence between CNNs and brain structure, and it is the reason people treat network depth as a candidate model of cortical depth rather than just a source of useful features.

8. Benchmarking: BrainScore, and the caveat that matters

There are now an enormous number of DNNs available, differing in architecture, training data and objective. Which best predict brain responses — which are most brain-aligned? Comparing across papers is hopeless when everyone uses their own stimuli, voxels and metrics, so the field built benchmarks for standardised comparison.

BrainScore The benchmark named in this lecture (brain-score.org). Fixed neural and behavioural datasets, fixed scoring, so any model can be submitted and placed on the same leaderboard.

Algonauts Project Public prediction challenges. The 2025 challenge was predicting fMRI responses while participants watched episodes of Friends; the 2027 challenge is predicting responses while participants play video games in the scanner.

In 2014, Yamins et al. ran the result that launched this whole enterprise: CNNs optimized for object categorisation were used to predict the brain, and the finding was a straight line — higher performance on the task went with higher prediction performance in the brain. The interpretation was seductive. Perhaps the brain's visual system is itself optimized for object categorisation, and any system that gets good at that task converges on the brain's solution.

Then the field kept making models better at ImageNet, and the line stopped holding.

State it plainly — this is exam-worthy High task performance does not guarantee high brain-prediction performance. As accuracy on ImageNet improved, models did not necessarily predict the brain better; by some measures DNNs have been getting less brain-like over time (Linsley et al., 2023). The inference to draw is about the objective: if being better at object categorisation stops making a model more brain-like, then object categorisation may not be all that vision evolved for. Vision also supports navigation, reaching and grasping, social inference, reading the lighting of a scene — the whole catalogue of things this course keeps finding specialised machinery for.

Try itYour new model is better and worse at the same time

You train two networks on the same images. Model A reaches 60 percent top-1 accuracy on object classification; Model B reaches 85 percent. You fit encoding models from each to the same V4 and IT voxels in the same participants, with the same ridge regression and the same held-out images. Model A explains more variance than Model B in both areas. A labmate concludes that the encoding analysis must be broken, since Model B is obviously the better vision system. What do you tell them?

Step 1 — What are we actually being asked?

Whether a dissociation between task performance and brain prediction is a bug or a finding. Note what is held constant: same stimuli, same voxels, same regression, same held-out split. The only thing varying is the model, so the comparison is a fair one.

Step 2 — Is there a reason to expect the two to track each other?

Only if object categorisation is the objective the brain is optimized for. That was the 2014 hypothesis, and the correlation Yamins et al. found was evidence for it. But it was always an empirical claim, not a definition — nothing guarantees that the features which squeeze out the last few points of ImageNet accuracy are features cortex uses. Models can win accuracy through tricks the brain has no use for: huge training sets, aggressive data augmentation, texture shortcuts, scale.

Step 3 — So what is the conclusion?

This is the Linsley et al. (2023) pattern, and it is a result rather than a failure. It tells you something about the objective function the brain is optimizing: whatever it is, it is not simply "categorise objects as accurately as possible". Follow it up by varying the objective deliberately rather than by chasing accuracy — which is exactly section 9.

One honest caveat before you publish: also check the noise ceiling, since no model can explain more variance than the data reliably contain, and confirm the result holds across participants.

Answer & the takeaway

Tell them the analysis is fine and that they have found the interesting thing. The problem was built around the assumption that "better at the task" means "more brain-like" — the single most common over-reading of the 2014 result. Benchmarks like BrainScore exist precisely so that brain prediction is scored separately from task accuracy, because the two come apart.

9. Varying architecture, data and objective

Once encoding models are your measuring instrument, every design choice in a network becomes a testable hypothesis about the brain. Richards et al. (2019) framed this as the deep learning framework for neuroscience: a model is specified by three things, and you can vary them one at a time.

What you vary	Examples from the lecture	What a difference in brain prediction would suggest
Architecture	CNNs versus vision transformers (ViT)	Whether the brain's computations depend on convolution and local receptive fields, or whether other wiring reaches the same representations
Training dataset	ImageNet versus LAION	Which statistics of the environment shape the representation — the data a system grows up on is the closest model analogue of visual experience
Objective function	Supervised, self-supervised, multimodal / language-supervised	What the system is being pressured to represent; the closest analogue of what vision is for

The architecture result is the humbling one. Vision transformers predict the brain well too (Conwell et al., 2024), even though they are not brain-inspired and have no convolution. Keep that in view whenever a good fit tempts you into a mechanistic claim.

Supervised, self-supervised, language-supervised

Supervised Every training image carries a human-provided label; the objective is to output the right one. Powerful, but needs millions of annotations and nothing like it exists for a developing infant.

Self-supervised No labels. The supervision comes from structure in the data itself — for contrastive methods like SimCLR, two distorted crops of the same photo must map to nearby representations while crops of different photos are pushed apart. The system learns what makes an image that image without being told what is in it.

Language-supervised The paired signal is text — a caption — rather than a category. SLIP combines self-supervised learning with this multimodal caption objective. The pressure is to represent whatever a sentence would mention.

Does language supervision improve brain alignment? Work by Aria Wang and colleagues at CMU (Wang et al., 2023) says yes, but not uniformly: language helps most in higher visual areas, which sit closer to semantic processing regions, and much less in early visual areas.

Why that pattern is the one to expect Early visual cortex is representing contrast, orientation and spatial frequency — things no caption mentions, and things any reasonable visual objective will produce. Higher visual cortex is representing what the scene is, which is exactly what a caption describes. So a training signal made of sentences should buy you little in V1 and a lot in IT. The gradient of the benefit is the evidence, and this is why the question is usually phrased as "early versus higher visual areas": a model's objective can be right about one end of the hierarchy and wrong about the other.

Visual development going deeper The midterm guide asks for an example of NeuroAI investigating visual development — this goes past the guest deck, so check it against the lecture recording. Human infants start with poor acuity and low contrast sensitivity: early visual input is effectively blurry, and sharpens over the first months. Researchers model this by training a network on blurred images first and sharpening them over training. Networks raised this way end up relying more on global shape and less on fine texture, and look more human in their recognition behaviour and in their robustness to changes in texture. The intuition is a developmental curriculum: fine texture is unavailable at the start, so the only features that work are large-scale configural ones, and those get built first and anchor everything after. Blurry vision looks like a bug in the hardware; as a training schedule it may be a feature.

10. Stimulus optimization: letting the brain pick the image

Here is the problem it solves. How do we know what a neural population cares about? Traditionally you hand-select stimuli — the localizer images from Stigliani et al. (2015) are a standard set — present a limited number of them, and compare categories to build a tuning function. Three limits follow immediately: you can only sample a small number of images, the approach is hypothesis-driven, and that introduces bias. You will never discover selectivity for something that was not on your list of categories.

Stimulus optimization An unbiased, data-driven alternative: rather than choosing images and asking which the region prefers, synthesise the image that maximises a target region's response — its "most exciting image". Because the search runs over everything the generator can draw, not over a predefined category list, it can discover properties of cortical organisation nobody proposed.

This needs a generator capable of making arbitrary realistic images, which is where diffusion models enter. A diffusion model generates images by iteratively removing noise: start from pure noise and repeatedly predict and subtract a little of it, and a coherent image emerges over many steps. A latent diffusion model runs this denoising in the compressed latent space of an autoencoder rather than on raw pixels, which is far less computationally expensive and can be conditioned on text (Rombach et al., 2022; Stable Diffusion is trained on about 2 billion images).

BrainDiVE Luo, Henderson, Wehbe & Tarr (2023). Couple a diffusion model to an encoding model of a target set of voxels, and steer the generation at each denoising step toward images the encoding model says those voxels will like. The encoding model is the fitness function; the diffusion model is the search. Out come synthetic images optimized for place-, face-, word-, body- and food-selective voxels. Henderson et al. (2026) then scanned 12 new participants on those images: model-generated images produced higher on-target responses in face-, body- and word-selective regions than the top-activating natural photographs did — "supra-activating" stimuli — and lower off-target responses in place-selective regions.

Two findings show why synthesis beats selection.

Food selectivity. Classical category-selective areas were found by picking categories — faces, places, words, bodies. Food was not on the list for decades. Recent work reports evidence for food-selective regions in higher visual cortex (Jain et al., 2023), and BrainDiVE images optimized for those voxels come out looking like food, which is independent support for the ROI. A predefined category list could not have found this; a generator searching over all images did.
Splitting apart two face areas. Images optimized for the occipital face area (OFA) contain more face parts; images optimized for the fusiform face area (FFA) contain more whole faces. Both regions are "face-selective" and a face-versus-object contrast cannot tell them apart, but the synthesised images make the difference visible, consistent with earlier versus later stages of face processing.

Stimulus optimization versus neural decoding They both pair generative models with brain data, and they are easy to confuse. Stimulus optimization synthesises an image to maximise the response of a target area: one brain area, across many trials, and the output is a preference. Neural decoding reconstructs the image that was shown on a given trial: many brain areas combined, one trial, and the output is a percept. Preference versus percept, many trials versus one.

11. What decoders are for: BCI, memory and imagery

Decoding started modestly. Mitchell et al. (2008) at CMU turned an encoding model into a decoder and could identify which of 60 nouns a person had read or viewed — above chance, from fMRI alone. Fifteen years later, Tang et al. (2023) used a language model (GPT-1) inside a Bayesian decoding framework to reconstruct the gist of continuous speech that people heard: not verbatim words, but sentences whose meaning often matched. The common structure is worth naming — propose candidate stimuli, score each by how well an encoding model says it predicts the observed brain response, keep the best.

Brain–computer interfaces

Brain–computer interface (BCI) Using a neural decoder's output to control a prosthetic or other device. The decoder stops being an analysis and becomes a control signal.

The lecture's example is Willett et al. (2021): a patient with microelectrode arrays implanted in the precentral gyrus (motor cortex) attempts to handwrite letters; a recurrent neural network decodes the intended letter from the recorded activity and types it. Note what makes this work where fMRI decoding could not — invasive electrodes give spiking activity at millisecond resolution, which a device you control in real time needs. The same approach decodes planned speech and planned hand movements, which is the basis of restoring communication to people who cannot speak or move.

Decoding memory and imagery

If visual cortex only responded to light arriving at the eye, there would be nothing to decode when the screen is blank. The sensory recruitment model of working memory says otherwise: the same visual areas are engaged when you see an item and when you remember it. Remembering is partly re-running the perceptual machinery.

That makes a prediction you can test with a decoder, and it holds (Harrison & Tong, 2009; Serences, Ester, Vogel & Awh, 2009):

In visual regions, you can decode what a participant imagines or remembers.
It works during the delay period, when no physical stimulus is present — so the decoded signal cannot be driven by the retinal image.
You can cross-decode: train the decoder on perception and test it on memory, and it still works. That is the strong evidence, because a classifier trained on seeing can only generalise to remembering if the two states share a representational format — a shared mechanism.

Say "decoding", not "mind-reading" The popular label oversells it badly. These decoders report which of a small set of orientations or categories is being held in mind, in a cooperative participant, using a model trained on hours of that specific person's data. "Mind-reading" implies arbitrary thoughts, from anyone, without consent. Keeping the distinction straight is what makes the ethics discussion below tractable rather than hysterical.

12. Challenges and ethics of brain decoding

The study guide asks for both, separately. Keep them apart: the first is about what does not work yet, the second about what happens if it does.

Challenges before reliable reconstruction of perceptual or cognitive content

The measurement is coarse and slow. Non-invasive decoding mostly uses fMRI, where each sample is a voxel-sized average of hundreds of thousands of neurons, delayed by seconds (section 3). Fast, fine-grained content is not in the data to be recovered.
It takes many hours per person. Models are fit per participant, often per voxel, on large bespoke datasets. Generalising across people is hard because cortical organisation differs anatomically and functionally — including small, variable tertiary sulci — and a decoder trained on one brain transfers poorly to another.
Reconstructions give gist, not detail. Decoded speech matches meaning more often than wording; decoded images match category, layout and texture better than identity. Every published example was also produced from a constrained stimulus set.
Generative priors can fill in what was never there. When a diffusion model or language model supplies most of the detail, a reconstruction can look convincing while the brain contributed only a weak constraint. Compelling pictures are a hazard in this field, not a result.
Prediction is not mechanism. Vision transformers predict well without being brain-like, and one voxel pattern is compatible with many neural codes — so brain alignment alone does not identify how the brain works.
Benchmarks are narrow. Scores depend on the stimulus set, the brain regions chosen and the noise ceiling of the data. Models can be tuned to a leaderboard rather than to the brain, and most datasets use static photographs in a scanner — not vision in the world, which is why the Algonauts challenges moved to films and games.

Ethical issues

Mental privacy. Thought has always been the one place no instrument could reach. Content decoded from imagery and memory is private in a way behaviour is not — and consider the obvious misuses: interrogation, insurance, employment screening, border control.
Consent, including the ability to withhold it. Today's decoders need the participant's cooperation during training, and a resistant participant can defeat them by thinking of something else. That is a real protection, and it is a property of current methods rather than a law of nature, so it should not be relied on permanently. Consent also has to cover what the data may later be used for: a dataset collected to study vision can be re-analysed with a decoder that did not exist when it was collected.
Accuracy cuts both ways. A decoder that works raises privacy problems; a decoder that half-works raises worse ones, if its output is treated as fact. Gist-level reconstructions presented as transcripts in a courtroom or a clinic would be the harm — overclaimed accuracy is an ethical issue, not merely a scientific one.
Clinical benefit is real, and asymmetric. BCIs restore communication to people who cannot speak and movement to people who are paralysed. A blanket restriction on decoding research carries a cost paid by exactly those patients, which is why the sensible proposals regulate use and data governance rather than banning the science.
Agency and identity in BCI. If a decoder completes your sentence or a device acts on a decoded intention you had not settled on, who authored the act? This matters most where a decoder has an error rate and the user cannot easily check its output.
Who holds the data. Neural recordings are identifiable, unchangeable and increasingly collected by consumer devices outside medical oversight. You cannot reissue a brain the way you reissue a password.

Takeaways Early AI models were inspired by the brain; today AI tools are used to understand the brain. Encoding models predict brain responses, decoding methods read information out, and stimulus optimization interprets the neural code by generating the input. NeuroAI now spans vision, semantic processing, brain–computer interfaces, recurrent networks and reinforcement learning — and is moving fast. (CMU offers Cognition in the Age of AI, 85-414, in spring.)

13. Check yourself

Why is predicting a voxel's response not the same as predicting a single neuron's activity?

Why: A 2 mm voxel is a cube of tissue containing on the order of several hundred thousand neurons, read out indirectly through blood flow and averaged over seconds. Option B reverses the scale — a voxel is enormously larger than a neuron, which is the whole problem. Option A inverts what BOLD measures: it is the hemodynamic signal that is recorded, and spiking that is inferred.

A network reaches state-of-the-art ImageNet accuracy but predicts IT responses worse than older, weaker models. The best interpretation is…

Why: This is the Linsley et al. (2023) result: task performance and brain prediction come apart, which is informative about the brain's objective rather than a methodological failure. Option C overshoots — IT clearly represents objects and scenes; what is in doubt is whether categorisation accuracy is the right pressure. Option D is too strong in the other direction: alignment does vary systematically with architecture, dataset and objective, which is exactly why those comparisons are worth running.

Which pairing of training objective and brain prediction matches the lecture's finding?

Why: Wang et al. (2023) compared SimCLR (self-supervised) with SLIP (self-supervised plus captions): language helps most in higher visual areas, closer to semantic processing regions. Option A is the inverted version and is implausible on reflection — captions do not describe contrast and orientation. Option C is wrong; self-supervised models are strong predictors, which is precisely why they are an interesting comparison.

What does stimulus optimization give you that choosing good photographs does not?

Why: Hand-picked stimuli are hypothesis-driven, few in number and biased toward categories you already thought of — which is how food selectivity was missed for decades. Option C is the decoding/stimulus-optimization confusion: decoding reconstructs one trial by combining many areas, while optimization finds one area's preference across many trials. Option B is a different method entirely — synthesising a stimulus still only measures responses, so causality needs perturbation.

Decoding orientation from visual cortex during a blank delay period, with a classifier trained on perception, supports which claim?

Why: Cross-decoding from perception to memory is the key design: a classifier trained on seeing can only transfer to remembering if the two share a representational format. Option D is what the result rules out. Option A is tempting because decoding feels fine-grained, but decoding works on distributed patterns of voxels, which is a different thing from single-neuron resolution.