DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment

Shentong Mo,Jing Shi,Yapeng Tian
2023-05-22
Abstract:Text-to-audio (TTA) generation is a recent popular problem that aims to synthesize general audio given text descriptions. Previous methods utilized latent diffusion models to learn audio embedding in a latent space with text embedding as the condition. However, they ignored the synchronization between audio and visual content in the video, and tended to generate audio mismatching from video frames. In this work, we propose a novel and personalized text-to-sound generation approach with visual alignment based on latent diffusion models, namely DiffAVA, that can simply fine-tune lightweight visual-text alignment modules with frozen modality-specific encoders to update visual-aligned text embeddings as the condition. Specifically, our DiffAVA leverages a multi-head attention transformer to aggregate temporal information from video features, and a dual multi-modal residual network to fuse temporal visual representations with text embeddings. Then, a contrastive learning objective is applied to match visual-aligned text embeddings with audio features. Experimental results on the AudioCaps dataset demonstrate that the proposed DiffAVA can achieve competitive performance on visual-aligned text-to-audio generation.
Computer Vision and Pattern Recognition,Machine Learning,Multimedia
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to synchronously align the generated audio with the visual content in the text - to - audio (TTA) generation task. Although existing methods can generate reasonable audio signals according to text descriptions, they often overlook the synchronization relationship between the audio and the video frames, resulting in the generated audio may not match the video frames. For example, when there is no train in the video frame, the model may generate a ten - second train whistle. To solve this problem, the paper proposes a new framework based on latent diffusion models (LDMs) - DiffAVA. By introducing a lightweight visual - text alignment module to update the text embedding, making it contain semantically visual - alignment information, thus achieving personalized TTA generation aligned with visual content. Specifically, DiffAVA utilizes a multi - head attention transformer to aggregate the temporal information in video features and uses a bimodal residual network to fuse the temporal visual representation with the text embedding. In addition, a contrastive learning objective is also adopted to match the text embedding with visual - alignment semantics with audio features. Experimental results show that the proposed DiffAVA achieves competitive performance in the visual - alignment TTA generation task.