M$^{2}$UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models

Shansong Liu,Atin Sakkeer Hussain,Chenshuo Sun,Ying Shan

2024-03-05

Abstract:The current landscape of research leveraging large language models (LLMs) is experiencing a surge. Many works harness the powerful reasoning capabilities of these models to comprehend various modalities, such as text, speech, images, videos, etc. They also utilize LLMs to understand human intention and generate desired outputs like images, videos, and music. However, research that combines both understanding and generation using LLMs is still limited and in its nascent stage. To address this gap, we introduce a Multi-modal Music Understanding and Generation (M$^{2}$UGen) framework that integrates LLM's abilities to comprehend and generate music for different modalities. The M$^{2}$UGen framework is purpose-built to unlock creative potential from diverse sources of inspiration, encompassing music, image, and video through the use of pretrained MERT, ViT, and ViViT models, respectively. To enable music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-modal understanding and music generation is accomplished through the integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA model to generate extensive datasets that support text/image/video-to-music generation, facilitating the training of our M$^{2}$UGen framework. We conduct a thorough evaluation of our proposed framework. The experimental results demonstrate that our model achieves or surpasses the performance of the current state-of-the-art models.

Sound,Multimedia,Audio and Speech Processing

What problem does this paper attempt to address?

The paper aims to address the problem of multimodal music understanding and generation, particularly in the context where research on utilizing large language models (LLMs) for music understanding and generation is still in its early stages. Specifically, the paper proposes a multimodal music understanding and generation framework called M2UGen, which integrates the capabilities of large language models to understand information from different modalities (such as text, images, and videos) and generate corresponding music. Additionally, the paper explores how to overcome the lack of training data by generating large multimodal music datasets and conducts a comprehensive evaluation of M2UGen, demonstrating its superior performance over current state-of-the-art models in various sub-tasks. Overall, the goal of the paper is to fill the gap in current research regarding multimodal music systems that combine understanding and generation capabilities.