Abstract:In movie productions, the Foley artist is responsible for creating an overlay soundtrack that helps the movie come alive for the audience. This requires the artist identify sounds that enhance the experience for the listener, reinforcing the director's intention for the scene. The artist must decide what artificial sound captures the essence of the sound and action depicted in the scene. In this paper, we present AutoFoley, an automated deep-learning tool that is used to synthesize a representative audio track for videos. AutoFoley is used to associate audio files with soundless video or to identify critical scenarios and provide a synthesized, reinforced and time-synchronized soundtrack. Our algorithm is capable of precise recognition of actions as well as interframe relations in fast- moving video clips through incorporating interpolation technique and temporal relational networks (TRN). We employ a robust multiscale recurrent neural network (RNN) and a convolutional neural network (CNN) for better understanding of the intricate input-to-output associations. To evaluate AutoFoley, we create an audio-video dataset containing a variety of sounds frequently used as Foley effects in movies. While the Foley dataset was limited to short-duration videos off the representative activities, this dataset demonstrates the capabilities of our proposed system. We show the synthesized sounds are portrayed with accurate temporal synchronization of the associated visual inputs. Human qualitative testing of AutoFoley shows more than 73% of the test subjects considered the generated soundtrack as original, which is a noteworthy improvement in comparable cross-modal research.

AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos With Deep Learning

FastFoley: Non-autoregressive Foley Sound Generation Based on Visual Semantics

FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds

Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound

FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos

Video-Guided Foley Sound Generation with Multimodal Controls

An Initial Exploration: Learning to Generate Realistic Audio for Silent Video

SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesis

Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis

Exploring Domain-Specific Enhancements for a Neural Foley Synthesizer

Conditional Generation of Audio from Video via Foley Analogies

Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

A Proposal for Foley Sound Synthesis Challenge

FOLEY-VAE: Generación de efectos de audio para cine con inteligencia artificial

Foley Sound Synthesis at the DCASE 2023 Challenge

FoleyGen: Visually-Guided Audio Generation

Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis

T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesis

Foley Music: Learning to Generate Music from Videos

AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection

AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition