Abstract:The Audio-Visual Video Parsing task aims to recognize and temporally localize all events occurring in either the audio or visual stream, or both. Capturing accurate event semantics for each audio/visual segment is vital. Prior works directly utilize the extracted holistic audio and visual features for intra- and cross-modal temporal interactions. However, each segment may contain multiple events, resulting in semantically mixed holistic features that can lead to semantic interference during intra- or cross-modal interactions: the event semantics of one segment may incorporate semantics of unrelated events from other segments. To address this issue, our method begins with a Class-Aware Feature Decoupling (CAFD) module, which explicitly decouples the semantically mixed features into distinct class-wise features, including multiple event-specific features and a dedicated background feature. The decoupled class-wise features enable our model to selectively aggregate useful semantics for each segment from clearly matched classes contained in other segments, preventing semantic interference from irrelevant classes. Specifically, we further design a Fine-Grained Semantic Enhancement module for encoding intra- and cross-modal relations. It comprises a Segment-wise Event Co-occurrence Modeling (SECM) block and a Local-Global Semantic Fusion (LGSF) block. The SECM exploits inter-class dependencies of concurrent events within the same timestamp with the aid of a new event co-occurrence loss. The LGSF further enhances the event semantics of each segment by incorporating relevant semantics from more informative global video features. Extensive experiments validate the effectiveness of the proposed modules and loss functions, resulting in a new state-of-the-art parsing performance.

Semantic Fusion Enhanced Event Detection Via Multi-Graph Attention Network with Skip Connection

Event Detection with Multi-Order Edge-Aware Graph Convolution Networks

EDSC: Event Detection with Superior Conceptual Information

Edge-Enhanced Graph Convolution Networks for Event Detection with Syntactic Relation

Event Detection Using a Self-Constructed Dependency and Graph Convolution Network

OntoED: Low-resource Event Detection with Ontology Embedding

Event Detection with Dual Relational Graph Attention Networks.

Event Detection with Relation-Aware Graph Convolutional Neural Networks.

Exploiting Semantic Space to Enhance Event Detection Combined with Event Knowledge

Event detection from text using path-aware graph convolutional network

MultiPLe: Multilingual Prompt Learning for Relieving Semantic Confusions in Few-shot Event Detection

Event Co-reference Disambiguation with Enhanced Semantics of Multi-layer Attention.

Event Detection With Dynamic Word-Trigger-Argument Graph Neural Networks

Event Detection As Graph Parsing

Chinese Event Detection Based on Multi-Feature Fusion and BiLSTM.

ES4ED: An Event Detection Model Based on Event Sentence Pre-Determination

Dependency Multi-Weight-view Graphs for Event Detection with Label Co-Occurrence

MGMP: Multimodal Graph Message Propagation Network for Event Detection

Exploiting Document-Level Information to Enhance Event Detection Combined with Semantic Space

Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing

Syntax-based dynamic latent graph for event relation extraction