Dance Generation by Sound Symbolic Words

Miki Okamura,Naruya Kondo,Tatsuki Fushimi,Maki Sakamoto,Yoichi Ochiai

2023-06-06

Abstract:This study introduces a novel approach to generate dance motions using onomatopoeia as input, with the aim of enhancing creativity and diversity in dance generation. Unlike text and music, onomatopoeia conveys rhythm and meaning through abstract word expressions without constraints on expression and without need for specialized knowledge. We adapt the AI Choreographer framework and employ the Sakamoto system, a feature extraction method for onomatopoeia focusing on phonemes and syllables. Additionally, we present a new dataset of 40 onomatopoeia-dance motion pairs collected through a user survey. Our results demonstrate that the proposed method enables more intuitive dance generation and can create dance motions using sound-symbolic words from a variety of languages, including those without onomatopoeia. This highlights the potential for diverse dance creation across different languages and cultures, accessible to a wider audience. Qualitative samples from our model can be found at: <a class="link-external link-https" href="https://sites.google.com/view/onomatopoeia-dance/home/" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Human-Computer Interaction,Sound,Audio and Speech Processing

What problem does this paper attempt to address?

The paper attempts to address the problem of enhancing creativity and diversity in dance creation by generating dance movements using onomatopoeia as input. Compared to traditional text- and music-based dance generation methods, onomatopoeia can express rhythm and meaning in an abstract vocabulary, free from language and cultural constraints, and without requiring specialized knowledge. The researchers propose a novel approach that utilizes the AI Choreographer framework and the Sakamoto system to extract features from onomatopoeia and combines them with the FACT model to generate dance movements. Additionally, they collected a dataset containing 44 onomatopoeias and corresponding dance movements, demonstrating that the method can successfully generate dance movements corresponding to the input onomatopoeia and is suitable for dance creation in various linguistic environments. Although the quality of the generated movements has not yet reached the state-of-the-art level, the study showcases the potential for diverse dance creation across languages and cultures and provides a more intuitive way of generating dance for a broad audience. Future research directions include improving the model, expanding the dataset, and developing interactive dance applications based on onomatopoeia.