Abstract:The audio-visual segmentation (AVS) task aims to segment sounding objects from a given video. Existing works mainly focus on fusing audio and visual features of a given video to achieve sounding object masks. However, we observed that prior arts are prone to segment a certain salient object in a video regardless of the audio information. This is because sounding objects are often the most salient ones in the AVS dataset. Thus, current AVS methods might fail to localize genuine sounding objects due to the dataset bias. In this work, we present an audio-visual instance-aware segmentation approach to overcome the dataset bias. In a nutshell, our method first localizes potential sounding objects in a video by an object segmentation network, and then associates the sounding object candidates with the given audio. We notice that an object could be a sounding object in one video but a silent one in another video. This would bring ambiguity in training our object segmentation network as only sounding objects have corresponding segmentation masks. We thus propose a silent object-aware segmentation objective to alleviate the ambiguity. Moreover, since the category information of audio is unknown, especially for multiple sounding sources, we propose to explore the audio-visual semantic correlation and then associate audio with potential objects. Specifically, we attend predicted audio category scores to potential instance masks and these scores will highlight corresponding sounding instances while suppressing inaudible ones. When we enforce the attended instance masks to resemble the ground-truth mask, we are able to establish audio-visual semantics correlation. Experimental results on the AVS benchmarks demonstrate that our method can effectively segment sounding objects without being biased to salient objects.

Improving Semantic Scene Categorization by Exploiting Audio-Visual Features

Human attention model for semantic scene analysis in movies

Towards Textually Describing Complex Video Contents with Audio-Visual Concept Classifiers

QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition

Detecting Violent Scenes in Movies by Auditory and Visual Cues.

Tag-based Semantic Features for Scene Image Classification

A Metadata Generation System with Semantic Understanding for Video Retrieval in Film Production

Audio scene semantic similarity computing approach

Audio-Visual Segmentation with Semantics

SemanticAC: Semantics-Assisted Framework for Audio Classification

A Statistics-Based Method For Video Semantic Analysis

Audiotory Movie Summarization by Detecting Scene Changes and Sound Events

A Study of Actor and Action Semantic Retention in Video Supervoxel Segmentation

Scene Segmentation Based on Video Structure and Spectral Methods

Movies2Scenes: Using Movie Metadata to Learn Scene Representation

Video Data Mining: Semantic Indexing and Event Detection from the Association Perspective

Rethinking movie genre classification with fine-grained semantic clustering

Learning a Contextual Multi-Thread Model for Movie/TV Scene Segmentation

Semantic Video Classification And Feature Subset Selection Under Context And Concept Uncertainty

Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics