Abstract:The audio-visual segmentation (AVS) task aims to segment sounding objects from a given video. Existing works mainly focus on fusing audio and visual features of a given video to achieve sounding object masks. However, we observed that prior arts are prone to segment a certain salient object in a video regardless of the audio information. This is because sounding objects are often the most salient ones in the AVS dataset. Thus, current AVS methods might fail to localize genuine sounding objects due to the dataset bias. In this work, we present an audio-visual instance-aware segmentation approach to overcome the dataset bias. In a nutshell, our method first localizes potential sounding objects in a video by an object segmentation network, and then associates the sounding object candidates with the given audio. We notice that an object could be a sounding object in one video but a silent one in another video. This would bring ambiguity in training our object segmentation network as only sounding objects have corresponding segmentation masks. We thus propose a silent object-aware segmentation objective to alleviate the ambiguity. Moreover, since the category information of audio is unknown, especially for multiple sounding sources, we propose to explore the audio-visual semantic correlation and then associate audio with potential objects. Specifically, we attend predicted audio category scores to potential instance masks and these scores will highlight corresponding sounding instances while suppressing inaudible ones. When we enforce the attended instance masks to resemble the ground-truth mask, we are able to establish audio-visual semantics correlation. Experimental results on the AVS benchmarks demonstrate that our method can effectively segment sounding objects without being biased to salient objects.

Audio Segmentation in AAC Domain for Content Analysis

Adaptive threshold method for real-time audio segmentation

An Effective Real-Time Audio Segmentation Method Based on Time-Frequency Energy Analysis

Audio Segmentation Based On Multi-Scale Audio Classification

Audio Segmentation Based on Size-Fixed Window and Layer Detection

A Two-Stage Content-Based Audio Segmentation Algorithm

An Automatic Approach Towards Audio Segmentation And Classification

Daubechies Wavelets Based Robust Audio Fingerprinting for Content-Based Audio Retrieval

Audio Segmentation Based on Wavelet Transform

A Front-end Tool of Content Analysis on MPEG Compressed Audio

Efficient Audio Stream Segmentation Via the Combined T-2 Statistic and Bayesian Information Criterion

Dual-Domain Audio Watermarking Algorithm Based on Flexible Segmentation and Adaptive Embedding

A robust audio fingerprinting algorithm in MP3 compressed domain

Segmentation-Based Audio Retrieval Algorithm

Audio scene semantic similarity computing approach

Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics

Compressed Domain Robust Hashing For Aac Audio

A Novel Classification-Based Audio Segmentation Algorithm

Inspection of Video Frequency Scene Based on Audio Frequency Analysis

Computationally Efficient Audio Segmentation Through a Multi-Stage BIC Approach

Object Segmentation with Audio Context