Abstract:In this paper we present three studies focusing on the effect of different sound models in interactive sonification of bodily movement. We hypothesized that a sound model characterized by continuous smooth sounds would be associated with other movement characteristics than a model characterized by abrupt variation in amplitude and that these associations could be reflected in spontaneous movement characteristics. Three subsequent studies were conducted to investigate the relationship between properties of bodily movement and sound: (1) a motion capture experiment involving interactive sonification of a group of children spontaneously moving in a room, (2) an experiment involving perceptual ratings of sonified movement data and (3) an experiment involving matching between sonified movements and their visualizations in the form of abstract drawings. In (1) we used a system constituting of 17 IR cameras tracking passive reflective markers. The head positions in the horizontal plane of 3-4 children were simultaneously tracked and sonified, producing 3-4 sound sources spatially displayed through an 8-channel loudspeaker system. We analyzed children's spontaneous movement in terms of energy-, smoothness- and directness-index. Despite large inter-participant variability and group-specific effects caused by interaction among children when engaging in the spontaneous movement task, we found a small but significant effect of sound model. Results from (2) indicate that different sound models can be rated differently on a set of motion-related perceptual scales (e.g., expressivity and fluidity). Also, results imply that audio-only stimuli can evoke stronger perceived properties of movement (e.g., energetic, impulsive) than stimuli involving both audio and video representations. Findings in (3) suggest that sounds portraying bodily movement can be represented using abstract drawings in a meaningful way. We argue that the results from these studies support the existence of a cross-modal mapping of body motion qualities from bodily movement to sounds. Sound can be translated and understood from bodily motion, conveyed through sound visualizations in the shape of drawings and translated back from sound visualizations to audio. The work underlines the potential of using interactive sonification to communicate high-level features of human movement data.

Perception of Animacy from the Motion of a Single Sound Object

Perceptual characterization of motion evoked by sounds for synthesis control purposes

Identifying critical kinematic features of animate motion and contribution to animacy perception

The Perception of Natural and Modulated Movement Sounds

Psychophysical evidence for auditory motion parallax

Perceiving Animacy with Causal Constraints: A "leash Resistance" Effect in Chasing Detection

Auditory motion affects visual biological motion processing

Filling-in visual motion with sounds

Auditory neuroscience: Sounds make the face move

Interactive Sonification of Spontaneous Movement of Children-Cross-Modal Mapping and the Perception of Body Movement Qualities through Sound

Contribution of self-motion perception to acoustic target localization

“Mickey Mousing” in the Brain: Motion-Sound Synesthesia and the Subcortical Substrate of Audio-Visual Integration

Self-motion reproduction can be affected by associated auditory cues

Perception of visual and audiovisual trajectories toward and away from the body in the first postnatal year

The Dynamics of Audiovisual Behavior in Speech

Active Listening for Spatial Orientation in a Complex Auditory Scene

Audio-visual interaction in visual motion detection: Synchrony versus Asynchrony

Capture of Intermodal Visual/Tactile Apparent Motion by Moving and Static Sound

Animacy and attentional processes: Evidence from the Stroop task

Increased body movement equals better performance? Not always! Musical style determines motion degree perceived as optimal in music performance