Bootstrapping Audio-Visual Segmentation by Strengthening Audio Cues

Tianxiang Chen,Zhentao Tan,Tao Gong,Qi Chu,Yue Wu,Bin Liu,Le Lu,Jieping Ye,Nenghai Yu
2024-02-06
Abstract:How to effectively interact audio with vision has garnered considerable interest within the multi-modality research field. Recently, a novel audio-visual segmentation (AVS) task has been proposed, aiming to segment the sounding objects in video frames under the guidance of audio cues. However, most existing AVS methods are hindered by a modality imbalance where the visual features tend to dominate those of the audio modality, due to a unidirectional and insufficient integration of audio cues. This imbalance skews the feature representation towards the visual aspect, impeding the learning of joint audio-visual representations and potentially causing segmentation inaccuracies. To address this issue, we propose AVSAC. Our approach features a Bidirectional Audio-Visual Decoder (BAVD) with integrated bidirectional bridges, enhancing audio cues and fostering continuous interplay between audio and visual modalities. This bidirectional interaction narrows the modality imbalance, facilitating more effective learning of integrated audio-visual representations. Additionally, we present a strategy for audio-visual frame-wise synchrony as fine-grained guidance of BAVD. This strategy enhances the share of auditory components in visual features, contributing to a more balanced audio-visual representation learning. Extensive experiments show that our method attains new benchmarks in AVS performance.
Computer Vision and Pattern Recognition,Sound,Audio and Speech Processing
What problem does this paper attempt to address?
The paper aims to address the issue of modality imbalance in the Audio-Visual Segmentation (AVS) task. Specifically: - **Problems with existing methods**: Most current AVS methods suffer from a unidirectional and insufficient fusion between audio cues and visual features, leading to visual features dominating the multimodal representation. This not only hinders the learning of joint audio-visual representations but also may result in inaccurate segmentation. - **Proposed method**: To tackle this issue, the authors propose a new framework named AVSAC. This framework includes two key components: - **Bidirectional Audio-Visual Decoder (BAVD)**: By establishing bidirectional bridges between two parallel decoder branches, it continuously strengthens audio cues and promotes deep interaction between audio and visual modalities, thereby alleviating the modality imbalance problem. - **Audio-Visual Frame-wise Synchrony (AVFS)**: Introducing a frame-wise synchrony loss function allows visual signals to learn from audio cues frame by frame, increasing the proportion of auditory components in visual features. In summary, the main goal of this paper is to achieve a balance between audio and visual information in the AVS task by enhancing audio cues, thereby improving segmentation accuracy.