C3LLM: Conditional Multimodal Content Generation Using Large Language Models

Zixuan Wang,Qinkai Duan,Yu-Wing Tai,Chi-Keung Tang

2024-05-25

Abstract:We introduce C3LLM (Conditioned-on-Three-Modalities Large Language Models), a novel framework combining three tasks of video-to-audio, audio-to-text, and text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities, synthesizing the given conditional information, and making multimodal generation in a discrete manner. Our contributions are as follows. First, we adapt a hierarchical structure for audio generation tasks with pre-trained audio codebooks. Specifically, we train the LLM to generate audio semantic tokens from the given conditions, and further use a non-autoregressive transformer to generate different levels of acoustic tokens in layers to better enhance the fidelity of the generated audio. Second, based on the intuition that LLMs were originally designed for discrete tasks with the next-word prediction method, we use the discrete representation for audio generation and compress their semantic meanings into acoustic tokens, similar to adding "acoustic vocabulary" to LLM. Third, our method combines the previous tasks of audio understanding, video-to-audio generation, and text-to-audio generation together into one unified model, providing more versatility in an end-to-end fashion. Our C3LLM achieves improved results through various automated evaluation metrics, providing better semantic alignment compared to previous methods.

Artificial Intelligence,Computation and Language,Machine Learning,Sound,Audio and Speech Processing

What problem does this paper attempt to address?

The paper attempts to address the problem of how to effectively combine information from video, audio, and text modalities in multimodal conditional generation tasks to produce high-quality and semantically aligned audio outputs. Specifically, the paper focuses on the following aspects:

1. **Fusion and Transformation of Cross-Modal Information**: How to effectively convert and generate information between different modalities (e.g., video to audio, audio to text, text to audio) to ensure that the generated content is not only of high quality but also maintains good semantic consistency with the input modalities.

2. **Quality and Diversity of Audio Generation**: How to generate high-quality audio using large language models (LLMs), especially when dealing with natural environmental sounds rather than music, ensuring that the generated audio has high fidelity and naturalness.

3. **Unified Framework for Multimodal Tasks**: How to construct a unified multimodal generation framework that allows the model to flexibly switch between various tasks, thereby improving the model's generality and adaptability.

To this end, the paper proposes C3LLM (Conditioned-on-Three-Modalities Large Language Model), a novel framework based on large language models that can handle tasks involving video to audio, audio to text, and text to audio. It enhances the quality and efficiency of audio generation through hierarchical audio encoders and non-autoregressive transformers.