Part VI · Hearing

Hearing & sound

The same inference problem as vision, in a different sense: the sound reaching your ear is the source multiplied by the room, and your brain has to undo the multiplication.

Essentials

Hearing at Marr's three levels, and six definitions you will be asked for.

Sound vs hearing — sound is the physical stimulus (air vibrations); hearing is the perceptual process. The falling tree makes sound with nobody there; it produces no hearing.
Computational level — identify objects, scenes and people, localise the speaker, enjoy music.
Algorithmic level — perceived sound = sound source × environment. One equation, two unknowns, exactly as in colour.
Implementation level — cochlea → auditory nerve → brainstem → thalamus → primary auditory cortex (A1). Each ear projects to both hemispheres.
Tonotopy — the spatial arrangement by which different frequencies are processed at different places in an auditory structure. A place code for frequency, the auditory counterpart of retinotopy. A1 runs high–low–high.
Transduction — mechanical vibration converted to electrical signal by hair cells in the cochlea; different hair cells answer to different frequencies.
Fundamental frequency — the lowest frequency a source produces; usually what you perceive as pitch. Harmonics — higher frequencies at whole-number multiples of the fundamental; their pattern is why a violin and a flute differ on the same note.
Vowels — produced without significant constriction of the vocal tract; carry characteristic formants, the frequency bands that give each vowel its identity. Consonants — involve constriction that obstructs airflow. Phonemes — the basic building blocks of speech; either vowels or consonants.
Spectrogram — frequency (vertical) against time (horizontal), with brightness showing how much energy sits at that frequency.
Sound constancy — recognising the same sound across rooms and speakers, because the brain has learned each one's statistical regularities and corrects for them.

Everything below is the explanation and the practice. Short on time: read this box, then go to the problems.

Two numbering systems — don't be thrown

The slide deck for this material is titled Lecture 13 — Hearing, Speech, Music, Language. The midterm study guide calls the same material Lecture 12: Hearing and Sound. They are the same lecture. This guide follows the study guide's numbering, because that is the document the exam is written against. The study guide also marks the speech-selective and music-selective regions as "featured in a later lecture" — so §6 covers them only lightly, and language is not on this midterm at all.

1. Hearing at Marr's three levels

The lecture is organised around Marr's three levels, and the exam asks you to apply them. Hearing is a clean case because the three levels come apart neatly.

Level	Question	For hearing
Computational	What problem, and why?	Identify objects and their properties, identify scenes, identify people, locate the speaker, enjoy music
Algorithmic	What representations and processes?	The sound you perceive depends on the source and the environment — so the system must separate them
Implementation	How is it realised in the brain?	Ear anatomy, the auditory pathway, primary auditory cortex (A1)

The five computational goals

Identify objects and their properties — a dropped coin versus a dropped book.
Identify scenes — a café, a stairwell, outdoors.
Identify people — recognising a voice before any word is understood.
Locate the speaker. The main cue is the interaural time difference: a sound off to one side arrives at the near ear slightly before the far ear. The difference is around 0.3 ms at the extremes — tiny, and the system resolves it reliably.
Enjoy music. The lecture's framing is worth remembering because it connects to §4: good music violates predictions with novel and unexpected elements. Pleasure in music is bound up with a prediction machine being surprised.

Definition — sound vs hearing

If a tree falls in a forest and nobody is around, does it make a sound? The lecture's answer is a definitional one. Sound is the external physical stimulus, air vibrations. Hearing is the perceptual process that lets us experience sound. So the tree makes sound and produces no hearing. Pitch belongs on the perceptual side too: it is the attribute that lets us judge a sound as high or low — perceived frequency, not frequency itself.

2. Reading a spectrogram

A spectrogram shows how the frequencies in a sound vary over time: time along the horizontal axis, frequency up the vertical, and brightness for how much energy sits at that frequency at that moment. A waveform shows you a single squiggle; the spectrogram decomposes it.

Figure 1 · A schematic vowel, labelled

Schematic, not measured data. The orange horizontal bars are the fundamental f₀ and its harmonics at 2f₀, 3f₀ and so on — evenly spaced because they are whole-number multiples. The two teal bands are formants: broad frequency regions carrying extra energy, set by the shape of the vocal tract. What to notice: the harmonics tell you the pitch; the formants tell you which vowel. Change the fundamental and you sing the same vowel higher; change the formants and you have said a different vowel at the same pitch.

The six definitions, precisely

Fundamental frequency — the lowest frequency produced by a source; often what we perceive as pitch.
Harmonics — higher frequencies that are multiples of the fundamental. Instruments sound different partly because they differ in which harmonics they emphasise.
Vowels — speech sounds produced without significant constriction or closure of the vocal tract.
Formants — the characteristic frequency bands that give a vowel its unique sound.
Consonants — speech sounds involving constriction of the vocal tract, obstructing airflow in various ways.
Phonemes — the basic building blocks of speech; they can be either vowels or consonants, and are often distinguished by subtle differences in mouth and tongue placement (sh vs s).

Two points the lecture makes about phonemes are easy exam fodder. First, consonant-to-vowel transitions matter: the same vowel preceded by different consonants produces different sounds, so speech is not a string of independent units. Second, phoneme inventories differ by language — English distinguishes r from l and Japanese does not; Hindi distinguishes dental from retroflex d and English does not. Which contrasts you can hear is shaped by the language you learned.

3. The algorithm: source × environment

The central equation

Perceived sound = sound source × environment. Sound reaches your ears directly from the source, and indirectly via reflections off surfaces — reverberation. Your ear receives the product. To recognise the source you must discount the room.

You have met this problem already

In colour the equation was perceived colour = illumination × surface reflectance, and the brain had to factor out the illuminant — which is why the dress and Adelson's checkerboard are ambiguous. Hearing poses the identical problem with environment in place of illumination. This is the most reusable idea in the course: the input underdetermines its cause, so perception is inference. If a question asks why audition needs prior experience at all, this is the answer.

How seriously should you take reverberation? In 1936 Stevens and Newman erected a swivel chair on top of a ventilator nine feet above the roof of Harvard's Biological Laboratories, because that appeared to be the only practicable way to avoid reflections contaminating their measurements. The room is not a nuisance you can ignore; it is multiplied into everything you hear.

Three versions of the same correction

The lecture works through three nuisance variables, and the solution has the same shape each time: the brain learns the statistical regularities and corrects for them.

Different rooms

Sound constancy. Your brain learns the reverberation characteristics — the impulse response — of different environments and corrects for them, so a voice is recognisably the same voice in a bathroom and a lecture hall.

Different speakers

Plot each English vowel as an ellipse of many talkers' productions and the ellipses overlap: the same vowel varies widely across talkers, and one acoustic sound can correspond to several different vowels. The brain learns each speaker's regularities. Hence the familiar voice advantage — speech from a familiar voice is perceived more accurately.

Competing sound

The cocktail party effect: the ability to focus on one sound while ignoring competing ones. Again, the brain has learned the statistics of the speaker and of the background and corrects. The lecture's conclusion is the thing to carry into the exam — your brain is not passively listening; it is actively generating predictions.

4. Prediction, and the Yanny/Laurel case

Rewrite the equation to make the inference explicit:

Perceived sound = sound source (ambiguous) × environment (inferred)

When the source is genuinely ambiguous, the brain predicts an environment, subtracts its presumed effect, and interprets what is left. Because the prediction can go either way, two listeners can hear the same recording differently — and the same listener can flip.

Why one clip yields two words

If the inferred environment is low frequency, the brain removes low frequencies, leaving the higher ones audible → "Yanny".
If the inferred environment is high frequency, the brain removes high frequencies, leaving the lower ones → "Laurel".

Note the logic: the percept is not a readout of the stimulus, it is the stimulus divided by a guess. Change the guess and the percept changes with no change in the input. That is the same move that makes the dress white-and-gold for some people and blue-and-black for others.

5. Implementation: cochlea to cortex

Transduction and tonotopy

The cochlea converts sound vibrations into neural signals. Transduction is that conversion — mechanical vibration to electrical signal by sensory hair cells — and it is the auditory counterpart of what opsins do for vision.

The key architectural fact: different hair cells are sensitive to different frequencies, arranged in order along the cochlea. That ordering is tonotopy — the spatial arrangement of where different frequencies are processed in an auditory structure. It is a place code: which neurons fire tells you which frequency arrived. Compare retinotopy, where position on the retina maps to position in cortex; here, frequency maps to position.

Figure 2 · The cochlea unrolled, and its frequency map

Schematic. Hair cells near the base respond to high frequencies (orange), those near the apex to low (teal). What to notice: frequency has been turned into place, and that map is preserved all the way up to cortex — which is what makes tonotopy worth a name.

The pathway

Figure 3 · Cochlea to primary auditory cortex

The lecture states the two endpoints — cochlea to primary auditory cortex — and one structural fact worth remembering: signals from each ear go to both sides of the brain (violet dashed). The intermediate brainstem and thalamic stages are drawn generically here, because the slide names the route rather than listing each nucleus; if your exam wants the individual nuclei, take them from the slide image rather than from this figure. A1 itself is tonotopic, running high–low–high across its surface.

Common mistake

The bilateral projection is a real contrast with vision. Damage to one side of the visual pathway produces a predictable field loss, because the projection is organised by visual field. One-sided damage in audition does not deafen one ear, because each ear already reaches both hemispheres. If a question asks why unilateral auditory cortex damage rarely causes deafness, that is the answer.

6. Speech, music and language later lecture

The deck continues past hearing, but the study guide marks the selective-regions question as featured later, and language is not on this midterm. Enough to recognise:

A speech (phoneme) selective region sits near A1; speech has been reconstructed from these regions using ECoG.
A music-selective region also sits near A1 — fMRI shows a stronger response to music than to non-musical sounds — with a nearby song-selective region.
Language relies on a network, mainly left frontal and left temporal, and the evidence that language is distinct from thought (global aphasia with intact cognition; Williams syndrome with low IQ and relatively intact language) is a double dissociation of the same form you met in the two visual pathways.

7. Problems

Try itTrace the pathway, and say what tonotopy buys you

A patient reports that speech sounds muffled and that they can no longer tell a flute from a violin, though they still hear that something is there. Trace the route a sound takes from air to cortex, then say which stage a loss of high-frequency hair cells would affect and why it could produce this complaint.

Step 1 — What is actually being asked?

Two things: the pathway in order, and a mapping from a cellular loss to a perceptual symptom. The second half is where the marks are — it tests whether you understand tonotopy as a code rather than as a word.

Step 2 — The pathway

Air vibration → cochlea, where hair cells transduce vibration into electrical signal → auditory nerve → brainstem nuclei → thalamus → primary auditory cortex (A1). Each ear projects to both hemispheres.

Step 3 — Where the loss lands

Hair cells are arranged tonotopically, high frequencies at the base. Losing high-frequency hair cells removes the top of the spectrum at the very first stage, before any central processing. Nothing downstream can restore information the cochlea never encoded.

Solution

Both symptoms follow. Muffled speech: consonants carry much of their identity in higher frequencies, so losing the top of the range blurs exactly the phonemic distinctions that separate words — while vowels, carried by lower formants, survive, so speech is still audible. Flute versus violin: instruments playing the same note share a fundamental and differ in their harmonics, which sit above it. Remove the high frequencies and you remove the difference, leaving the pitch intact.

Why this is the exam version of the idea

It rewards knowing that the fundamental carries pitch while harmonics and upper formants carry identity. A student who can only recite "tonotopy = frequency map" cannot get from a hair-cell loss to a specific complaint.

Try itWhy does the same clip yield two different words?

Two people listen to the same recording on the same speaker. One hears "Yanny", the other "Laurel". Explain in terms of the algorithmic-level equation, and say what this tells you about whether perception is a readout of the stimulus.

Step 1 — Write the equation

Perceived sound = sound source (ambiguous) × environment (inferred). Both listeners receive the same left-hand side. They differ in the second factor — the environment they infer.

Step 2 — Work both branches

If the inferred environment is low frequency, the brain discounts low frequencies; what remains is the higher content → "Yanny". If it is inferred as high frequency, high frequencies are discounted and the lower content remains → "Laurel".

Solution & the takeaway

The percept is the stimulus divided by a guess about the room, so two different guesses yield two different words from identical input. Perception is therefore not a readout — it is inference, and the brain is actively generating predictions rather than passively listening. The same argument explains the dress in colour: identical photons, different assumed illuminant, different percept.

Try itSound constancy across rooms and speakers

Your friend says one word to you in a tiled bathroom and then outdoors. The acoustic signal reaching your ear differs substantially. You hear the same word both times. Explain the mechanism, then explain the separate problem posed by a different person saying that word, and name the empirical phenomenon that supports your answer.

Step 1 — Separate the two nuisance variables

Room and speaker are different factors multiplied into the same signal. The bathroom changes the environment term (reverberation); a new talker changes the source term.

Step 2 — The room

Sound constancy. The brain has learned the statistical regularities of reverberation — effectively the impulse response of rooms like this one — and corrects for it. This is structurally identical to colour constancy inferring the illuminant.

Step 3 — The speaker

Harder, because talkers genuinely overlap: plotting each vowel as an ellipse over many speakers, the ellipses overlap, so one acoustic sound can correspond to several vowels. The brain learns each speaker's regularities and corrects for them.

Solution — and the evidence

Both are solved by learning the statistics of a nuisance factor and discounting it. The supporting phenomenon is the familiar voice advantage: listeners perceive speech more accurately from a familiar voice than an unfamiliar one — exactly what you expect if recognition depends on a learned speaker model. The cocktail party effect is the same story with background noise as the nuisance.

8. Check yourself

A singer sings the same vowel first at a low pitch, then an octave higher. On the spectrogram, what mainly changes?

Why: pitch lives in the fundamental, so f₀ and all its multiples move. Formants are set by the shape of the vocal tract, not by pitch, so they stay put — which is exactly why the vowel is still recognisable as the same vowel. Swap which one changes and you have the two definitions backwards, the single most common error on this material.

Why does damage to auditory cortex on one side rarely produce deafness in one ear?

Why: the auditory pathway is bilateral from early on, so one hemisphere still receives both ears. This is a genuine contrast with vision, where the projection is organised by visual field and one-sided damage produces a predictable field loss.

A tree falls in an empty forest. On the lecture's definitions, what happens?

Why: sound is defined as the external physical stimulus — air vibrations — and hearing as the perceptual process that experiences it. The vibrations occur; nobody perceives them. The question is testing whether you hold the physical/perceptual distinction, the same distinction that separates frequency from pitch.