Abstract:Speech sounds convey a great deal of information about the scenes, resulting in a variety of effects ranging from reverberation to additional ambient sounds. In this paper, we manipulate input speech to sound as though it was recorded within a different scene, given an audio-visual conditional example recorded from that scene. Our model learns through self-supervision, taking advantage of the fact that natural video contains recurring sound events and textures. We extract an audio clip from a video and apply speech enhancement. We then train a latent diffusion model to recover the original speech, using another audio-visual clip taken from elsewhere in the video as a conditional hint. Through this process, the model learns to transfer the conditional example's sound properties to the input speech. We show that our model can be successfully trained using unlabeled, in-the-wild videos, and that an additional visual signal can improve its sound prediction abilities. Please see our project webpage for video results: https://tinglok.netlify.app/files/avsoundscape/

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to use self - supervised learning methods to make the input speech sound as if it was recorded in different scenarios. Specifically, the author proposes a new audio - visual soundscape stylization problem. Given a visual or audio - visual example from a specific scenario and a clean input speech, the goal is to manipulate this input speech so that it sounds as if it was recorded in this scenario while retaining the acoustic properties and ambient sounds of this scenario. ### Specific description of the problem 1. **Limitations of existing methods**: - Existing audio generation methods mainly focus on reproducing the acoustic characteristics of the room (such as reverberation), but ignore other factors that affect the recording effect, such as ambient sounds like wind and bird calls. - These methods usually require simulated data or labeled data for training, which limits their scalability and practical application scope. 2. **Goals of the paper**: - Propose a self - supervised learning - based method that can train the model using only unlabeled in - the - wild video data. - The model can not only capture the acoustic characteristics of the scenario (such as reverberation), but also capture ambient sounds (such as the sound of sea waves, wind, etc.), so that the input speech more realistically reflects the sound characteristics of the target scenario. ### Overview of the solution The author proposes a method based on the conditional diffusion model, which is achieved through the following steps: 1. **Data preparation**: - Randomly extract two non - overlapping segments from the video, one as a conditional example and the other as the target audio. - Separate and enhance the target audio, removing the scenario - specific properties (such as background noise and reverberation). 2. **Model training**: - Use the conditional example as a cue to train the model to restore the enhanced target audio to its original state. - Through self - supervised learning tasks, the model learns to transfer the acoustic and environmental properties in the conditional example to the input speech. 3. **Testing phase**: - During testing, provide the model with a conditional example from a different scenario, and the model will adjust the style of the input speech according to this example to make it conform to the sound characteristics of the target scenario. ### Main contributions - **Self - supervised learning**: The model can be trained using only in - the - wild video data without labels. - **Multi - modal fusion**: Combine audio and visual signals to improve the model's understanding and transfer ability of sound characteristics. - **Wide applicability**: It is applicable not only to speech, but also to non - speech input sounds. Through this method, the paper shows how to effectively transfer the acoustic and environmental properties of the scenario to the input speech, thereby achieving more realistic sound stylization.

Self-Supervised Audio-Visual Soundscape Stylization

Learning Visual Styles from Audio-Visual Associations

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

Audio-Visual Contrastive Learning with Temporal Self-Supervision

Conditional Generation of Audio from Video via Foley Analogies

SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model

Self-Supervised Video Forensics by Audio-Visual Anomaly Detection

Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision

Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment

Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning

Curriculum Audiovisual Learning

Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generation

Contrastive Conditional Latent Diffusion for Audio-visual Segmentation

SonicVisionLM: Playing Sound with Vision Language Models

AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement

Weakly Supervised Visual-Auditory Saliency Detection with Multigranularity Perception.

Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation

Self-supervised Learning of Audio Representations from Audio-Visual Data using Spatial Alignment

The Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning

STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment