Beyond colour perception: auditory synaesthesia elicits visual experience of colour, shape, and spatial location

Auditory-visual synaesthesia, a rare condition in which sounds evoke involuntary visual experiences, provides a window into how the brain normally combines audition and vision. Previous research primarily focuses on synaesthetic colour, but little is known about other synaesthetic visual features. Here we tested a group of synaesthetes for whom sounds elicit visual experience of 'geometric objects' comprising colour, shape, and spatial location. In an initial session, we presented sounds and asked synaesthetes to draw their synaesthetic experiences. Changes in auditory pitch and timbre affect synaesthetic experience in a manner similar to the cross-modal correspondences of non-synaesthetes (high-pitched sounds are associated with brighter, smaller, and spatially higher objects). To objectively measure these experiences, we devised a crossmodal multi-feature synaesthetic interference paradigm. Synaesthete participants performed colour/shape discriminations. The results show mismatches between display images and synaesthetic features can significantly slow reaction times. Moreover, voluntary attention modulates cross-modal interference of synaesthetic features: Attending to one feature reduces the impact of another mismatching feature. Our findings go beyond the typical focus on colour perception by showing shape and location are integral parts of visual synaesthetic experience. The similarity between auditory-visual synaesthesia and normal cross-modal correspondences implies they rely on the same cognitive/neural mechanisms.