Abstract:In this paper we present three studies focusing on the effect of different sound models in interactive sonification of bodily movement. We hypothesized that a sound model characterized by continuous smooth sounds would be associated with other movement characteristics than a model characterized by abrupt variation in amplitude and that these associations could be reflected in spontaneous movement characteristics. Three subsequent studies were conducted to investigate the relationship between properties of bodily movement and sound: (1) a motion capture experiment involving interactive sonification of a group of children spontaneously moving in a room, (2) an experiment involving perceptual ratings of sonified movement data and (3) an experiment involving matching between sonified movements and their visualizations in the form of abstract drawings. In (1) we used a system constituting of 17 IR cameras tracking passive reflective markers. The head positions in the horizontal plane of 3-4 children were simultaneously tracked and sonified, producing 3-4 sound sources spatially displayed through an 8-channel loudspeaker system. We analyzed children's spontaneous movement in terms of energy-, smoothness- and directness-index. Despite large inter-participant variability and group-specific effects caused by interaction among children when engaging in the spontaneous movement task, we found a small but significant effect of sound model. Results from (2) indicate that different sound models can be rated differently on a set of motion-related perceptual scales (e.g., expressivity and fluidity). Also, results imply that audio-only stimuli can evoke stronger perceived properties of movement (e.g., energetic, impulsive) than stimuli involving both audio and video representations. Findings in (3) suggest that sounds portraying bodily movement can be represented using abstract drawings in a meaningful way. We argue that the results from these studies support the existence of a cross-modal mapping of body motion qualities from bodily movement to sounds. Sound can be translated and understood from bodily motion, conveyed through sound visualizations in the shape of drawings and translated back from sound visualizations to audio. The work underlines the potential of using interactive sonification to communicate high-level features of human movement data.

Cross-linguistically shared and language-specific sound symbolism in novel words elicited by locomotion videos in Japanese and English

Takete and Maluma in Action: A Cross-Modal Relationship between Gestures and Sounds

Guessing Meaning From Word Sounds of Unfamiliar Languages: A Cross-Cultural Sound Symbolism Study

Japanese Sound-Symbolic Words for Representing the Hardness of an Object Are Judged Similarly by Japanese and English Speakers

Iconicity mediates semantic networks of sound symbolism

Symbouki: a meta-analysis on the emergence of sound symbolism in early language acquisition

Dance Generation by Sound Symbolic Words

With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models

Phonetic underpinnings of sound symbolism across multiple domains of meaning

Exploring the nature of cumulativity in sound symbolism: Experimental studies of Pokémonastics with English speakers

Sound symbolism in Japanese names: Machine learning approaches to gender classification

What voiced obstruents symbolically represent in Japanese: evidence from the Pokémon universe

The typology of sound symbolism: Defining macro-concepts via their semantic and phonetic features

English Speakers Can Infer Pokémon Types Based on Sound Symbolism

Assessing sound symbolism: Investigating phonetic forms, visual shapes and letter fonts in an implicit bouba-kiki experimental paradigm

Visualizing Video Sounds With Sound Word Animation to Enrich User Experience

Interactive Sonification of Spontaneous Movement of Children-Cross-Modal Mapping and the Perception of Body Movement Qualities through Sound

Brain networks underlying the processing of sound symbolism related to softness perception

Neural Basis of the Sound-Symbolic Crossmodal Correspondence Between Auditory Pseudowords and Visual Shapes

Bouba/Kiki in Touch: Associations Between Tactile Perceptual Qualities and Japanese Phonemes

Sound symbolism processing is lateralized to the right temporal region in the prelinguistic infant brain