Part IV · Vision

Colour, V4 & the two pathways

Why colour is an inference the brain has to un-multiply, how three cones and two opponent channels deliver it, what cortical damage takes away that retinal damage doesn't — and the double dissociation that splits vision into "what" and "where".

Essentials

1. Colour at Marr's three levels

Start with the thing that makes colour worth having. A ripe berry and an unripe one are the same shape and size; a peacock's tail carries no information at all in greyscale. Colour is how a visual system separates objects that geometry cannot separate — and the two uses the lecture names are the two that pay in evolutionary currency: finding food and finding mates.

Newton's notebook (1661) supplies the physics: white light decomposes into wavelengths, and the band we see runs from about 400 to 700 nm. But wavelength is not colour. Colour is what the brain builds, and the reason it's hard is the next line.

LevelColour perception
Computational (why?)Finding food, finding mates, recognising objects — separating surfaces that differ in material but not in shape
Algorithmic (how?)A light source illuminates a surface; the surface reflects some wavelengths. The light reaching the eye is the product of the two: perceived colour = illumination × reflectance. To recover the object's colour the brain must divide the illumination back out — colour constancy
Implementation (where?)Cones (S, M, L) in the retina → centre-surround retinal ganglion cells with colour opponency → V4. The visual hierarchy now reads V1 edges, V2 surfaces, V1–V5 depth, V5/MT motion, V4 colour
Slide typo worth noticing The L10 slide that lays out these three levels is titled "Depth perception across Marr's three levels" — a leftover heading from Lecture 8. The content underneath is colour. If the exam asks for colour at Marr's three levels, the rows above are what it wants.

2. One equation, two unknowns

The three quantities Illumination — the light source (noon sun, overcast sky, tungsten bulb, sodium streetlamp). Surface reflectance — the fraction of each wavelength a surface sends back when lit by white light; this is the property that belongs to the object. Perceived colour — the light that hits your eye, which depends on both.

The retina measures only the product. Write it as a single number per channel and the problem becomes obvious: the eye receives \(r = i \times \rho\), where \(i\) is illumination and \(\rho\) is reflectance. The brain wants \(\rho\) — a berry's redness shouldn't change at dusk. It has one measurement and two unknowns. Every colour illusion on this page is that arithmetic refusing to have a unique answer.

Analogy You are told a rectangle's area is 27 and asked for its height. You can't answer — unless you assume a width. Assume 30 and the height is 0.9; assume 90 and it's 0.3. The retina hands the brain the area; the brain supplies a width from context and then reports the height as if it had measured it. Where the analogy stops: the assumed width isn't arbitrary — shadows, nearby surfaces and the overall light in the scene drive it.
The same retinal signal, two different worlds
What to notice: the light arriving at the eye is held fixed at 0.27 — the left patch never changes. Drag the slider and only the interpretation moves: assume dim light (0.30) and the surface must be reflecting 0.90, a white card in shadow; assume bright light (0.90) and the same signal means reflectance 0.30, a dark grey card in sun. Both are exactly consistent with the measurement, because 0.9 × 0.3 and 0.3 × 0.9 are both 0.27. The right-hand panel redraws the inferred surface; grey levels are drawn by layering --ink at varying opacity, so they track the theme.

3. Colour constancy, and the two canonical illusions

Colour constancy Taking the light that hits the eye as input, the brain corrects it using its prediction about the illumination, so that a surface keeps roughly the same apparent colour as the lighting changes. Without it, a red apple would look brown indoors and you'd have to relearn every object's colour every time a cloud moved.

Constancy is a feature, and the illusions are the feature being caught in the act. Both canonical examples work the same way: the retinal signal is fixed, the assumed illumination differs across two regions, so the inferred reflectance differs — and inferred reflectance is what you consciously see.

Adelson's checkerboard

A checkerboard sits under a cylinder that casts a shadow. Square A is on the lit part of the board; square B is inside the shadow. A and B send identical light to your eye — identical pixel values. They look nothing alike: B looks clearly lighter.

The brain's reasoning, written out: the light from B equals the light from A; but B is in shadow, so B is being illuminated less; to return the same amount of light under less illumination, B's surface must be a lighter colour. That inference is correct physics applied to a trick photograph. Divide a fixed numerator by a smaller denominator and the quotient goes up.

Adelson's checkerboard — reconstruction
What to notice: A and B are painted with the same value (0.46) in both conditions — A is a dark square in the light, B a light square (0.78) dimmed by the shadow to 0.78 × 0.59 = 0.46. Press "Connect A and B" to draw a bar of that value between them: it is continuous, which is the proof they match. Untick the shadow and nothing about A or B changes in the drawing — only the illumination story does, and with no illumination difference to explain away there is nothing to divide out. A reconstruction, not Adelson's photograph, so expect a weaker effect than the slides.

The dress

The 2015 photograph that split the internet. Some people see white and gold, others blue and black, from the same pixels. The photograph is badly exposed and gives almost no reliable cue to the light source — so the illumination term is genuinely ambiguous, and different viewers plug in different values.

Nobody is seeing it wrong. Two observers running the same correct algorithm on the same data with different priors about the lighting must reach different answers — that is what "illumination is ambiguous" means, and it's the sentence to write on the exam.

Where the slide mislabels a term The checkerboard slide reads "A and B are the same pixel values (reflectance: color hitting your eye)". That parenthesis swaps two terms. The light hitting your eye is perceived colour — the product illumination × reflectance. Reflectance is the surface property, which is precisely the thing A and B do not share in your percept. Getting these two the right way round is most of the credit on a short-answer version of this question.
Try itSame signal, opposite conclusions

Two matte cards, photographed in one scene. Both send the same amount of light to the camera: a value of 0.27 on a scale where 1.0 is the brightest possible. Card A sits in a shadow where the illumination is 0.30. Card B sits in open light where the illumination is 0.90.

(a) What reflectance does each card have? (b) Which card is physically the lighter-coloured card? (c) An observer shown the photograph with the shadow cropped out reports that the two cards look the same. Are they wrong?

Step 1 — What are we actually being asked?

Not "which is brighter in the image" — that's given and identical. We're asked to invert perceived colour = illumination × reflectance for the surface property. Three quantities, and the question moves between them: the retinal signal (0.27, shared), the illumination (differs), the reflectance (to find).

Step 2 — Do the division

Reflectance = signal ÷ illumination. A: 0.27 ÷ 0.30 = 0.90. B: 0.27 ÷ 0.90 = 0.30. Multiply back: 0.90 × 0.30 = 0.27 and 0.30 × 0.90 = 0.27. Both reproduce the measurement, which is the whole difficulty.

Answer & the takeaway

(a) A is 0.90, B is 0.30. (b) A — reflectance 0.90 is a near-white card, B at 0.30 is dark grey. (c) The observer isn't wrong about the image; the patches really are identical there. Cropping removed the cue they needed to estimate illumination, so they can't run the division and fall back on reporting the raw signal. Constancy didn't fail — its input was removed.

The confusion this is built around: treating "the colour hitting your eye" and "the colour of the surface" as one quantity. They're a product and a factor, and the checkerboard, the dress and achromatopsia's spared cones all turn on keeping them apart.

4. Trichromacy: three numbers for a whole spectrum

In the 1850s Helmholtz asked people to match colours using adjustable lights. With two lights they could not match everything; with three they could match everything; four added nothing but difficulty. That is a behavioural measurement of a biological fact — the retina has three cone types — made decades before anyone could see them.

The three cones S cones — short wavelengths, peak ~420 nm, "blue". M cones — medium, peak ~530 nm, "green". L cones — long, peak ~560 nm, "red". Each is broadly tuned: a single cone's firing rate is ambiguous between wavelength and intensity, which is why one cone type can never signal colour on its own. Trichromatic theory: perceived colour varies with the ratio of responses across the three.
Cone response curves — drag a wavelength through them
What to notice: the curves overlap heavily, and M and L almost completely — their peaks are only about 30 nm apart. The readout computes the three activations and the L:M ratio live: 0.88 at 530 nm (M ahead), 2.31 at 620 nm (L well ahead). That ratio is the hue signal, and 550 vs 560 nm differ by barely 0.1 in it — which is why fine red–green judgements go first when a cone is missing. Tick "Remove L cones" and the ratio becomes unavailable: 530 and 620 nm are now the same measurement. The curves are a schematic reconstruction (Gaussians at 420/530/560 nm), not digitised physiology — peaks and M–L overlap accurate, exact shapes not.

Colour addition follows directly. If hue is a ratio across three channels, then any ratio can be produced by mixing three suitably chosen lights — which is exactly what the screen you are reading this on does. A television has no yellow pixels; it drives your L and M cones in the ratio that 580 nm light would produce, and you see yellow. You are not seeing the wavelength. You are seeing the ratio.

Colour blindness

Colour blindness — more precisely a colour-vision deficiency — comes from losing or altering a cone type. Drop the L cones and every pair of wavelengths that differed only in how much more they excited L than M becomes indistinguishable: red and green collapse. The affected person is not seeing grey everywhere; they are seeing a world with one fewer dimension, and hues that were far apart for you land on the same point for them.

Where the deck and the study guide disagree The deck's trichromacy slide says perceived colour varies with the ratio of the three cone responses. The L10 study guide says "sum up all the activity in the cones to perceive certain colours". Ratio is the defensible version, and the figure above shows why: the sum of cone activity rises when a light gets brighter without its wavelength changing at all, so a sum confounds colour with intensity. Safest exam answer: colour is coded by the relative activity across S, M and L — summing or comparing across the three cone types, not reading any one of them alone.
Why a sodium streetlamp ruins colour A sodium lamp emits essentially one narrow band of yellow light. Reflectance only matters for wavelengths that are present: a surface cannot send back light it was never given. Under illumination with one wavelength, two cars with very different reflectance spectra return light that differs only in amount, not in composition — so the cone ratios are near identical and the colours become hard to tell apart. This is the illumination × reflectance equation again: make the illumination zero across most of the spectrum and most of the reflectance information is multiplied away before the eye ever sees it.
Try itWhich cone is missing?

A participant is shown pairs of single-wavelength lights and asked whether they look the same or different. Their results:

  • 530 nm vs 620 nm — "the same"
  • 440 nm vs 530 nm — "different"
  • 440 nm vs 620 nm — "different"

(a) Which cone class is most likely absent? (b) Is this deficit retinal or cortical, and what single extra test would tell you? (c) Why does the participant still see a difference in the other two pairs?

Step 1 — What are we actually being asked?

To reason from a pattern of confusions back to a missing channel. The useful move is to ask, for each pair, which cone comparison normally distinguishes those two wavelengths — then find the comparison that the "same" judgement has knocked out. Part (b) is a different question entirely: not which channel, but at which stage of the pathway.

Step 2 — Work out which comparison is dead

530 nm and 620 nm are both long enough that S cones barely respond to either (0.007 and essentially 0 in the figure above). What normally separates them is the L:M ratio — 0.88 versus 2.31. If those two look identical, the L-versus-M comparison has been lost, so either L or M is missing. The pairs involving 440 nm are still discriminable, which tells us S is intact and is doing that work.

Answer & the takeaway

(a) L or M — the red–green channel (losing L is protanopia, losing M deuteranopia); the data as given don't distinguish which. (b) Retinal. The signature is that the confusions are selective and probably lifelong: specific pairs collapse, everything else is normal. The one extra test: ask whether colour was lost suddenly and whether all hue is gone, including in memory and imagery — acquired, total loss with intact cones means cortex (next section). (c) S cones are intact, so any short-versus-long comparison still works and 440 nm still differs from both.

The transferable point: a missing cone removes a dimension of colour space, not colour itself, and which pairs collapse tells you which dimension went.

5. Colour opponency in retinal ganglion cells

Cones hand on to retinal ganglion cells, and those cells do not simply relay cone activity. They are centre-surround — the same architecture that gives luminance contrast in Chapter 10 — but wired across cone types. A cell might be excited by red in its centre and inhibited by green in its surround. Its output no longer says "there is red light here"; it says "there is more red here than around here".

Colour opponency The visual system encodes colour using opposing neural channels, built by subtracting cone signals from one another: roughly red-versus-green and blue-versus-yellow. Excitation by one member of a pair is inhibition by the other, so a single channel's output is a signed comparison, not an absolute level.
Why subtract? Two reasons. Colour contrast: comparing a region with its surround is what makes the boundary between two surfaces visible, and boundaries carve a scene into objects. And it helps constancy: the illumination term multiplies a whole neighbourhood roughly equally, so a centre-versus-surround comparison partially cancels it. The division problem gets easier if the front end already computes ratios, not levels.
A sloppy phrase in the deck The slide defines opponency as "red vs. green cones, blue vs. yellow cones". There is no yellow cone — there are only S, M and L. The yellow signal is constructed by adding L and M, and the blue-yellow channel is then S versus (L + M). Say "opponent channels built from cone comparisons", not "yellow cones", and the point survives scrutiny.

The lecture's exit ticket, answered

The deck asks: why might a grey background appear green after you stare at a pink or purple image? Try it, then read the mechanism.

Colour adaptation — stare at the centre for 15 seconds
What to notice: keep your eyes on the small marker. When the violet patch is replaced by a neutral --bg-soft grey, the grey takes on a faint greenish cast for a few seconds — the complement of what you adapted to. The bars on the right are the mechanism: two opponent populations whose balance sets the percept. The adapting patch drives the violet side hard, that side fatigues, and when neutral grey arrives — which should drive both sides equally — the unfatigued side wins. The effect is deliberately subtle here because the page uses only its own palette (--accent-3 violet, --accent-2 teal) rather than saturated primaries; the slides' purple-and-green demo is the stronger version of the same thing.
The answer Perceived colour depends on the balance between opponent channels, and a neutral grey is neutral precisely because it leaves that balance even. Staring at purple keeps one side firing hard, and sustained firing fatigues it. Now put up a grey that should produce a tie: the fatigued side responds below normal while its opposite responds normally, the balance tips, and the brain reports the opposite colour. Same logic as the motion aftereffect (Ch. 12) and tilt adaptation (Ch. 10). That the aftereffect is always the complement is the evidence that these channels are genuinely opponent rather than independent.

6. Cerebral achromatopsia

Cerebral achromatopsia Loss of colour vision caused by damage to the brain, commonly V4. The eyes and cones are intact; the deficit is cortical. The lecture's case is Mr. I, the colourblind painter: after a head injury his world became, in his description, grey and leaden — flesh looked rat-coloured, food looked repellent. Within two months his drawing skill was undiminished; what had gone was colour.

Set the two disorders side by side, because the contrast is the question:

Retinal colour blindnessCerebral achromatopsia
DamageCone photoreceptors, in the retinaCortex, commonly V4
OnsetUsually congenital, lifelongAcquired — a stroke or injury, with a before and after
What is lostOne dimension: specific hue pairs collapse (e.g. red–green)Colour as a whole — the scene goes grey
ConesMissing or alteredIntact and responding normally
Colour in memory, dreams, imageryWas never there to loseCan also be lost — Mr. I could not even imagine colour

Why this matters beyond terminology: Mr. I's cones are fine, so trichromatic machinery alone is not sufficient for colour experience — something downstream must be intact too, and the lesion site says where. That is Chapter 5's logic, a selective loss localising a function, and it is the lecture's strongest evidence that V4 does the job.

7. Dorsal and ventral: "where" and "what"

Past V4 the hierarchy stops being a single ladder and splits. The same V1 output feeds two streams that ask different questions about the same image.

The two streams on a brain schematic (left hemisphere, lateral view)
V1 occipital DORSAL — "where" parietal · position, visually guided action lesion → optic ataxia, simultanagnosia VENTRAL — "what" temporal / IT · meaning, recognition lesion → object agnosia (DF) frontal ↑ dorsal (toward the top of the head) ↓ ventral (toward the base of the brain)
What to notice: the two streams leave the same occipital source and are named for nothing more than their direction — dorsal (teal) runs up and forward to parietal cortex, ventral (violet) runs down and forward through temporal cortex. The outline is a schematic, not a traced anatomical image. Full feedforward route: eye → LGN → V1 → V2 → V4 → IT.
Dorsal — the "where" pathway

Occipital → parietal. Codes the position of visual stimuli and the spatial properties an action needs: where a thing is, how it is oriented, how to reach it. Vision for action.

Ventral — the "what" pathway

Occipital → temporal (inferior temporal cortex, IT). Codes the meaning of visual stimuli: identity, category, recognition. Vision for perception.

The first evidence was monkey lesion studies: parietal lesions impaired tasks requiring the spatial location of a stimulus, while temporal lesions impaired tasks requiring which object it was. Human patient work made it sharper — next section.

What the ventral stream is made of

IT is a hierarchy of feature detectors with category-selective regions sitting at the top. These recur across the course, so learn them as a set with their lesion syndromes:

RegionPrefersLesion gives
LOC — lateral occipital complexObject shape; responds more to intact than scrambled objects, whereas early visual cortex responds about equallyObject agnosia
FFA — fusiform face areaFaces — or, on Tarr's account, fine-grained discrimination among similar-looking things, which is why it also responds in car, bird and greeble expertsImpaired face recognition
PPA — parahippocampal place areaScenes and the geometry of surrounding spaceTopographical agnosia — cannot recognise familiar places and landmarks
VWFA — visual word form areaWritten words in the reader's own scriptImpaired word reading

VWFA carries an extra idea worth having ready. Writing is a few thousand years old — far too recent for a dedicated cortical area to have evolved for it. Neuronal recycling: circuits that evolved for one function get recruited for a newer cultural skill. The deck's other example is arithmetic, where ECoG electrodes over parietal cortex — the dorsal "where" stream — show strong activity during number processing. Spatial circuitry, repurposed for maths.

Dorsal lesions have a second syndrome besides optic ataxia: simultanagnosia, the inability to see more than one object at a time. Note how it differs from agnosia — each object is recognised fine; what fails is holding several in a spatial layout at once, which is a "where" problem, not a "what" problem.

8. The double dissociation, done properly

This is the reasoning pattern the course keeps returning to, and this is the cleanest instance of it. The general machinery is in the dissociations chapter; here is the vision version.

Patient DF — the ventral half

DF has a lesion in the ventral visual pathway (LOC). Her profile:

That is object agnosia: an inability to recognise the shape of objects, with vision, memory and intelligence otherwise working. Drawing from memory but failing to recognise her own drawing rules out a loss of object knowledge and localises the failure to getting from image to identity.

Now the part that makes DF famous. Ask her to judge the orientation of a slot — a perceptual report — and she is at chance. Hand her a card and ask her to post it through the slot, and her hand rotates to the correct orientation on the way. Same slot, same visual information, two uses: visually guided action intact, perceptual judgement not. The lecture's clarinet patient shows the same split — he can't recognise it, but he picks it up and handles it correctly.

The deck's unanswered question: "3 limitations" The slide asks what's wrong with stopping here and answers only "it is a single dissociation: 3 limitations". The three, spelled out: All three share a structure: they are explanations in terms of a single graded resource rather than two separable systems.

Optic ataxia — the dorsal half

Optic ataxia Difficulty grasping objects because of impaired spatial localisation, following a dorsal stream lesion. These patients have intact object recognition — they can tell you exactly what the object is — but their reach-and-grasp is inaccurate, and their hand does not preshape to the object's orientation.

Put the two patients in a table and the argument completes itself:

PatientLesionRecognise the object ("what")Grasp / post it ("where")
DFVentral (LOC)ImpairedIntact
Optic ataxiaDorsal (parietal)IntactImpaired
Why two patients, and what it licenses One patient gives a single dissociation: task A impaired, task B spared. Every single dissociation has the same escape route — that A is just harder, or more vulnerable to general injury, than B. A double dissociation closes it. If a second patient shows the opposite profile — B impaired, A spared — then "A is harder than B" cannot be true, because it would have to predict A impaired in both. No ordering of the two tasks by difficulty can produce a crossed pattern. The conclusion you are then licensed to draw: recognition and visually guided action depend on at least partly separable systems. What you are not licensed to draw: that the two streams never interact, or that each region does only this one thing. A double dissociation establishes separability, not isolation or anatomical purity.
One more control worth naming With an object agnosia patient, also test tactile recognition and knowledge of the object's function. If DF can identify by touch an object she cannot identify by sight, the deficit is specifically visual — not a loss of object concepts, and not a naming problem.
Try itBuild the double dissociation from scratch

You have two patient groups — Group V with ventral occipitotemporal lesions, Group D with dorsal parietal lesions — plus matched controls. You want to establish that visual recognition and visually guided action are separable.

(a) Name two tasks that use identical visual input but differ in which process they demand. (b) State the exact pattern of results that would support a double dissociation. (c) Both groups turn out to be impaired on both tasks. What explanations remain, and how would you change the experiment?

Step 1 — What are we actually being asked?

Not "what do the streams do" — that's the hypothesis. We're asked to design a test of it, which means the stimulus must be held constant while the demanded process varies. If the two tasks use different stimuli, any difference between them could be a stimulus effect and the design is dead before data collection.

Step 2 — Pick the two tasks, and the pattern you need

The slot-and-card pair is the model. Task 1 (recognition): report the slot's orientation, verbally or by rotating a handheld card to match without reaching. Task 2 (action): post the card through it, scored by hand orientation at insertion. Identical slot, lighting and viewing time; orientation varied trial to trial so neither task is solvable from memory.

The supporting pattern is crossed. Group V: impaired on the report, at control level on posting. Group D: at control level on the report, impaired on posting. Controls off both floor and ceiling on both tasks, so the measure has room to show impairment either way. A double dissociation is a crossover — not just a group difference, but each group failing the task the other handles.

Answer & the takeaway

(c) If both groups fail both tasks you have no dissociation, and three alternatives stay live: both tasks too hard (a floor effect), lesions large enough to hit both streams, or general injury effects swamping any selective deficit. Fixes: make both tasks easier until controls are comfortably off the floor; check lesion extent on imaging and exclude participants whose damage crosses both streams; add a task neither stream should affect as a general-impairment control; titrate both tasks to the same control accuracy before comparing.

The step students skip: a double dissociation is not "two patients" and not "two tasks" — it is the crossing, because no single difficulty dimension can produce a crossing. Carry that test to the aphasias, to rhythm versus melody, and to anything else in this course claiming two separable processes.

9. Check yourself

A patch of a photograph reflects light to your eye with a value of 0.27. You see it as a white surface. What has the brain assumed?
Why: Reflectance = signal ÷ illumination. To get a high reflectance (white) out of a fixed signal, the assumed illumination must be low — a white card in shadow. Option A inverts the conclusion with the assumption; option C gives you a dark surface instead; option D is the error the whole chapter exists to correct, since 0.27 is a product of two unknowns.
A painter reports after a stroke that his world has become grey and leaden, that food looks repellent, and that he can no longer even imagine colour. Ophthalmic examination finds normal cones. The most likely diagnosis is…
Why: Normal cones plus acquired, total loss of colour including imagery means the damage is cortical, not retinal — the dissociation that defines cerebral achromatopsia. A cone deficiency would be congenital and would collapse specific hue pairs rather than removing colour wholesale, and it could never affect imagined colour. The two dorsal/ventral syndromes are distractors from the wrong half of the lecture.
Patient DF cannot report the orientation of a slot but posts a card through it accurately. On its own, what does this establish?
Why: This is one patient with one spared and one impaired task, so the difficulty explanation is still open; you need the optic ataxia patient with the crossed profile before option A is earned. Option B reverses the streams, and option C contradicts the ventral lesion that produced her agnosia.
Which statement about trichromacy and opponency is correct?
Why: They are complementary, not rival: three cone types capture the spectrum, then ganglion cells subtract those signals into opponent channels to code contrast. Option C invents a fourth cone — the error the deck's wording invites. Option D gets the anatomy backwards: trichromacy is the receptor-level account.