Abstract:Audio-visual question answering (AVQA) requires reference to video content and auditory information, followed by correlating the question to predict the most precise answer. Although mining deeper layers of audio-visual information to interact with questions facilitates the multimodal fusion process, the redundancy of audio-visual parameters tends to reduce the generalization of the inference engine to multiple question-answer pairs in a single video. Indeed, the natural heterogeneous relationship between audiovisuals and text makes the perfect fusion challenging, to prevent high-level audio-visual semantics from weakening the network's adaptability to diverse question types, we propose a framework for performing mutual correlation distillation (MCD) to aid question inference. MCD is divided into three main steps: 1) firstly, the residual structure is utilized to enhance the audio-visual soft associations based on self-attention, then key local audio-visual features relevant to the question context are captured hierarchically by shared aggregators and coupled in the form of clues with specific question vectors. 2) Secondly, knowledge distillation is enforced to align audio-visual-text pairs in a shared latent space to narrow the cross-modal semantic gap. 3) And finally, the audio-visual dependencies are decoupled by discarding the decision-level integrations. We evaluate the proposed method on two publicly available datasets containing multiple question-and-answer pairs, i.e., Music-AVQA and AVQA. Experiments show that our method outperforms other state-of-the-art methods, and one interesting finding behind is that removing deep audio-visual features during inference can effectively mitigate overfitting. The source code is released at <a class="link-external link-http" href="http://github.com/rikeilong/MCD-forAVQA" rel="external noopener nofollow">this http URL</a>.

Learning to Answer Questions in Dynamic Audio-Visual Scenarios

AVQA: A Dataset for Audio-Visual Question Answering on Videos

Question-Aware Global-Local Video Understanding Network for Audio-Visual Question Answering

Boosting Audio Visual Question Answering via Key Semantic-Aware Cues

Multi-Granularity Relational Attention Network for Audio-Visual Question Answering

Progressive Spatio-temporal Perception for Audio-Visual Question Answering

Heterogeneous Interactive Graph Network for Audio-Visual Question Answering

Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering

Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios

Answering Diverse Questions via Text Attached with Key Audio-Visual Clues

Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering

SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answering

CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios

Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering

SAVEn-Vid: Synergistic Audio-Visual Integration for Enhanced Understanding in Long Video Context

Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation

Audio-Visual Segmentation

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

Video Question Answering via Knowledge-based Progressive Spatial-Temporal Attention Network

VQA$^2$:Visual Question Answering for Video Quality Assessment

CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answering