Abstract:We are witnessing a revolution in conditional image synthesis with the recent success of large scale text-to-image generation methods. This success also opens up new opportunities in controlling the generation and editing process using multi-modal input. While spatial control using cues such as depth, sketch, and other images has attracted a lot of research, we argue that another equally effective modality is audio since sound and sight are two main components of human perception. Hence, we propose a method to enable audio-conditioning in large scale image diffusion models. Our method first maps features obtained from audio clips to tokens that can be injected into the diffusion model in a fashion similar to text tokens. We introduce additional audio-image cross attention layers which we finetune while freezing the weights of the original layers of the diffusion model. In addition to audio conditioned image generation, our method can also be utilized in conjuction with diffusion based editing methods to enable audio conditioned image editing. We demonstrate our method on a wide range of audio and image datasets. We perform extensive comparisons with recent methods and show favorable performance.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper attempts to address the problem of how to use audio signals to guide image generation and editing. Specifically, the researchers propose a new method called **SonicDiffusion**, which introduces audio conditions into a pre-trained diffusion model to generate images that are semantically aligned with the audio content. ### Background and Motivation Currently, text-based image generation and editing techniques (such as DALL·E 2, Latent Diffusion Models, and Imagen) have achieved significant results, but these methods rely on text input, which may not be completely consistent with visual content. In contrast, there is a more direct and natural association between audio signals and visual scenes, which can provide rich contextual information. Therefore, integrating audio signals into the image generation process can produce more immersive and realistic images. ### Main Contributions 1. **New Diffusion Model**: SonicDiffusion extends the pre-trained Stable Diffusion model to generate images based on audio signals. 2. **Audio-Image Cross-Attention Layers**: Introduces audio-image cross-attention layers, which require only a small number of trainable parameters to achieve the fusion of audio signals with image generation. 3. **Image Editing Functionality**: Supports not only audio-driven image generation but also audio-driven image editing, modifying existing images through feature injection. ### Method Overview 1. **Audio Projector Module**: Converts audio clips into a series of internal space tokens, which are then integrated into the image generation model. The audio projector module is trained using a combination of Mean Squared Error (MSE) and contrastive loss functions to ensure the alignment of audio tokens with text tokens in the semantic space. 2. **Gated Cross-Attention Layers**: Introduces new gated cross-attention layers in the decoder of the diffusion model, allowing image features to dynamically interact with audio tokens. The gating mechanism helps maintain the denoising efficiency of the model. 3. **Audio-Guided Image Editing**: Achieves audio-guided image editing by injecting the residuals and self-attention features generated in reverse from the source image into subsequent image generation stages. ### Experimental Results The researchers conducted extensive tests on multiple datasets, and the results show that SonicDiffusion outperforms existing methods in both audio-to-image generation and audio-driven image editing. Particularly in generating landscape images, the model can capture high-frequency details and accurately render facial features influenced by sound. ### Conclusion SonicDiffusion expands the capabilities of pre-trained diffusion models by introducing audio signals as conditions, achieving richer and more multimodal image generation and editing. This method not only excels in generation quality but also demonstrates the great potential of audio in enhancing image generation and editing.

SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models

AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation

Diffusion Model-Based Image Editing: A Survey

Improving Diffusion-Based Image Synthesis with Context Prediction

A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation

DiffBlender: Scalable and Composable Multimodal Text-to-Image Diffusion Models

Collaborative Diffusion for Multi-Modal Face Generation and Editing

Pix2Video: Video Editing using Image Diffusion

Controlled and Conditional Text to Image Generation with Diffusion Prior

Align, Adapt and Inject: Sound-guided Unified Image Generation

Sketch-Guided Text-to-Image Diffusion Models

CreativeSynth: Creative Blending and Synthesis of Visual Arts based on Multimodal Diffusion

Modulating Pretrained Diffusion Models for Multimodal Image Synthesis

Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis

Are Diffusion Models Vision-And-Language Reasoners?

PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor

BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion