UniMuMo: Unified Text, Music and Motion Generation

Han Yang,Kun Su,Yutong Zhang,Jiaben Chen,Kaizhi Qian,Gaowen Liu,Chuang Gan

2024-10-07

Abstract:We introduce UniMuMo, a unified multimodal model capable of taking arbitrary text, music, and motion data as input conditions to generate outputs across all three modalities. To address the lack of time-synchronized data, we align unpaired music and motion data based on rhythmic patterns to leverage existing large-scale music-only and motion-only datasets. By converting music, motion, and text into token-based representation, our model bridges these modalities through a unified encoder-decoder transformer architecture. To support multiple generation tasks within a single framework, we introduce several architectural improvements. We propose encoding motion with a music codebook, mapping motion into the same feature space as music. We introduce a music-motion parallel generation scheme that unifies all music and motion generation tasks into a single transformer decoder architecture with a single training task of music-motion joint generation. Moreover, the model is designed by fine-tuning existing pre-trained single-modality models, significantly reducing computational demands. Extensive experiments demonstrate that UniMuMo achieves competitive results on all unidirectional generation benchmarks across music, motion, and text modalities. Quantitative results are available in the \href{<a class="link-external link-https" href="https://hanyangclarence.github.io/unimumo_demo/" rel="external noopener nofollow">this https URL</a>}{project page}.

Sound,Computer Vision and Pattern Recognition,Graphics,Machine Learning,Multimedia,Audio and Speech Processing

What problem does this paper attempt to address?

The paper aims to address the issue of a unified framework for cross-modal generation tasks. Specifically, its goal is to perform generation tasks involving any combination of the three modalities: Music, Motion, and Text. The paper proposes a unified multimodal model named UniMuMo, which can generate content in other modalities given any one modality as input. For example, it can generate synchronized music and motion from text, or generate motion from music and text descriptions.

The paper mainly addresses the following two key challenges:
1. **Data Pairing Issue**: Due to the lack of comprehensive datasets containing all three modalities (music, motion, and text), the authors align unpaired music and motion sequences based on rhythm patterns and use a dynamic time warping algorithm to adjust motion sequences to match the music rhythm.
2. **Unified Architecture Design Issue**: To support conditional generation tasks for all three modalities, the authors propose a novel joint encoding scheme that maps motion to the same feature space as music and introduces a music-motion parallel generation scheme, enabling all generation tasks to be implemented within a single transformer decoder architecture.

Through these methods, UniMuMo can handle not only unidirectional generation tasks (such as text to music, music to motion, etc.) but also more complex combination generation tasks (such as text to music plus motion). Additionally, the model demonstrates competitiveness on all unidirectional generation benchmarks, validating its effectiveness and generality.