Abstract:The visual perception of individuals is thought to be mediated by a network of regions in occipitotemporal cortex that supports specialized processing of faces, bodies, and actions. In comparison, we know relatively little about the neural mechanisms that support the perception of multiple individuals and the interactions between them. The present study sought to elucidate the visual processing of social interactions by identifying which regions of the social perception network represent interpersonal synchrony. In an fMRI study with 32 human participants (26 female, 6 male), we used multi-voxel pattern analysis to investigate whether activity in face-selective, body-selective, and interaction-sensitive regions across the social perception network supports decoding of synchronous vs. asynchronous head-nodding and head-shaking. Several regions were found to support significant decoding of synchrony/asynchrony, including extrastriate body area, face-selective and interaction-sensitive mid/posterior right superior temporal sulcus, and occipital face area. We also saw robust cross-classification across actions in extrastriate body area, suggestive of movement-invariant representations of synchrony/asynchrony. Exploratory whole-brain analyses also identified a region of right fusiform cortex that responded more strongly to synchronous than to asynchronous motion. Critically, perceiving interpersonal synchrony/asynchrony requires the simultaneous extraction and integration of dynamic information from more than one person. Hence, the representation of synchrony/asynchrony cannot be attributed to augmented or additive processing of individual actors. Our findings therefore provide important new evidence that social interactions recruit dedicated visual processing within the social perception network that extends beyond that engaged by the faces and bodies of the constituent individuals. Significance statement The presence of interpersonal synchrony is a critical cue when appraising the nature and content of social interactions from third-person perspectives. However, little is known about its representation within the human visual system. Here, we use fMRI to reveal distributed representations of interpersonal synchrony/asynchrony in several regions of the social perception network, notably extrastriate body area and superior temporal sulcus. There is growing speculation that the perception of social interactions engages specialized visual processing beyond that recruited by the faces and bodies of the constituent individuals. Critically, perceiving interpersonal synchrony requires the simultaneous extraction and integration of dynamic information from more than one person. These results therefore provide key new evidence of dedicated multi-actor processing within the social perception network.

Auditory dyadic interactions through the “eye” of the social brain: How visual is the posterior STS interaction region?

A multimodal interface for speech perception: the role of the left superior temporal sulcus in social cognition and autism

Processing communicative facial and vocal cues in the superior temporal sulcus

Auditory, Visual and Audiovisual Speech Processing Streams in Superior Temporal Sulcus

Mouth and voice: A relationship between visual and auditory preference in the human superior temporal sulcus

Representational interactions during audiovisual speech entrainment: Redundancy in left posterior superior temporal gyrus and synergy in left motor cortex

Modulation of Brain Activity by Selective Attention to Audiovisual Dialogues

A Double Dissociation between Anterior and Posterior Superior Temporal Gyrus for Processing Audiovisual Speech Demonstrated by Electrocorticography

Inter-Brain Network Underlying Turn-Based Cooperation and Competition: A Hyperscanning Study Using Near-Infrared Spectroscopy

Attentional and linguistic interactions in speech perception

Multisensory integration of speech and gestures in a naturalistic paradigm

Interaction effect of reproducing spatialized human-induced and environmental sounds from nearby sources on the social Simon effect

Unraveling dyadic psycho-physiology of social presence between strangers during an audio drama - a signal-analysis approach

Multisensory Integration Sites Identified by Perception of Spatial Wavelet Filtered Visual Speech Gesture Information

Convergence of spoken and written language processing in the superior temporal sulcus

Retinotopic effects during spatial audio-visual integration

How Auditory Experience Differentially Influences the Function of Left and Right Superior Temporal Cortices

The Causal Role of Left and Right Superior Temporal Gyri in Speech Perception in Noise: A Transcranial Magnetic Stimulation Study

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution

Concurrent talking in immersive virtual reality: on the dominance of visual speech cues

Neural representations of observed interpersonal synchrony/asynchrony in the social perception network