Abstract:The Audio-Visual Video Parsing task aims to identify and temporally localize the events that occur in either or both the audio and visual streams of audible videos. It often performs in a weakly-supervised manner, where only video event labels are provided, \ie, the modalities and the timestamps of the labels are unknown. Due to the lack of densely annotated labels, recent work attempts to leverage pseudo labels to enrich the supervision. A commonly used strategy is to generate pseudo labels by categorizing the known video event labels for each modality. However, the labels are still confined to the video level, and the temporal boundaries of events remain unlabeled. In this paper, we propose a new pseudo label generation strategy that can explicitly assign labels to each video segment by utilizing prior knowledge learned from the open world. Specifically, we exploit the large-scale pretrained models, namely CLIP and CLAP, to estimate the events in each video segment and generate segment-level visual and audio pseudo labels, respectively. We then propose a new loss function to exploit these pseudo labels by taking into account their category-richness and segment-richness. A label denoising strategy is also adopted to further improve the visual pseudo labels by flipping them whenever abnormally large forward losses occur. We perform extensive experiments on the LLP dataset and demonstrate the effectiveness of each proposed design and we achieve state-of-the-art video parsing performance on all types of event parsing, \ie, audio event, visual event, and audio-visual event. We also examine the proposed pseudo label generation strategy on a relevant weakly-supervised audio-visual event localization task and the experimental results again verify the benefits and generalization of our method.

Reinforced Label Denoising for Weakly-Supervised Audio-Visual Video Parsing

Revisit Weakly-Supervised Audio-Visual Video Parsing from the Language Perspective

DHHN: Dual Hierarchical Hybrid Network for Weakly-Supervised Audio-Visual Video Parsing

Label-anticipated Event Disentanglement for Audio-Visual Video Parsing

Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling

Improving Audio-Visual Video Parsing with Pseudo Visual Labels

Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing

Audio-Visual Speech Separation with Visual Features Enhanced by Adversarial Training

Toward a Perceptive Pretraining Framework for Audio-Visual Video Parsing

Cross-Modal learning for Audio-Visual Video Parsing

Distributed Audio-Visual Parsing Based on Multimodal Transformer and Deep Joint Source Channel Coding.

Multi-Level Signal Fusion for Enhanced Weakly-Supervised Audio-Visual Video Parsing

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

Vision-Language Models are Strong Noisy Label Detectors

AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition

Robust Audio-Visual ASR with Unified Cross-Modal Attention

Active Label-Denoising Algorithm Based on Broad Learning for Annotation of Machine Health Status

Multimodal Deep Denoise Framework for Affective Video Content Analysis.

Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser

XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception