Abstract:The Audio-Visual Video Parsing task aims to recognize and temporally localize all events occurring in either the audio or visual stream, or both. Capturing accurate event semantics for each audio/visual segment is vital. Prior works directly utilize the extracted holistic audio and visual features for intra- and cross-modal temporal interactions. However, each segment may contain multiple events, resulting in semantically mixed holistic features that can lead to semantic interference during intra- or cross-modal interactions: the event semantics of one segment may incorporate semantics of unrelated events from other segments. To address this issue, our method begins with a Class-Aware Feature Decoupling (CAFD) module, which explicitly decouples the semantically mixed features into distinct class-wise features, including multiple event-specific features and a dedicated background feature. The decoupled class-wise features enable our model to selectively aggregate useful semantics for each segment from clearly matched classes contained in other segments, preventing semantic interference from irrelevant classes. Specifically, we further design a Fine-Grained Semantic Enhancement module for encoding intra- and cross-modal relations. It comprises a Segment-wise Event Co-occurrence Modeling (SECM) block and a Local-Global Semantic Fusion (LGSF) block. The SECM exploits inter-class dependencies of concurrent events within the same timestamp with the aid of a new event co-occurrence loss. The LGSF further enhances the event semantics of each segment by incorporating relevant semantics from more informative global video features. Extensive experiments validate the effectiveness of the proposed modules and loss functions, resulting in a new state-of-the-art parsing performance.

Multi-Level Signal Fusion for Enhanced Weakly-Supervised Audio-Visual Video Parsing

Specialty may be better: A decoupling multi-modal fusion network for Audio-visual event localization

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

DHHN: Dual Hierarchical Hybrid Network for Weakly-Supervised Audio-Visual Video Parsing

Toward a Perceptive Pretraining Framework for Audio-Visual Video Parsing

Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing

Audio-Visual Event Localization Via Recursive Fusion by Joint Co-Attention

Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing

Exploring Audio-Visual Information Fusion for Sound Event Localization and Detection In Low-Resource Realistic Scenarios

Mixed in Time and Modality: Curse or Blessingƒ Cross-Instance Data Augmentation for Weakly Supervised Multimodal Temporal Fusion

Audio-Visual Information Fusion Using Cross-Modal Teacher-Student Learning for Voice Activity Detection in Realistic Environments

Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection

CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing

Multi-Modulation Network for Audio-Visual Event Localization

Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition

Audio-Visual Speech Recognition Using A Two-Step Feature Fusion Strategy.

Unimodal-Multimodal Collaborative Enhancement for Audio-Visual Event Localization.

MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition

Distributed Audio-Visual Parsing Based on Multimodal Transformer and Deep Joint Source Channel Coding.

Audio-Visual Fusion Layers for Event Type Aware Video Recognition