Abstract:Audio-visual question answering (AVQA) is an emerging task that aims to provide answers by integrating visual contents, audio streams, and their associations within given videos. The major challenge lies in effectively fusing heterogeneous multi-modal data to comprehend complex scenes while capturing question-related clues to infer correct answers. Current AVQA models primarily employ attention mechanisms to extract questionrelated clues separately from visual and audio modalities before combining them. However, these approaches have two limitations: (1) They neglect the exploration of the association and complementary between audio and visual; (2) Encoding visual or audio holistically limits the capacity to capture the cross-modal and crosstemporal dynamic events. In this paper, we introduce the Heterogeneous Interactive Graph Network, a novel solution designed to address these limitations. Specifically, we construct heterogeneous multi-modal graphs that facilitate unified integration of multiple modalities, including visual, audio, and question. This approach effectively explores the associations and complementarity among multiple modalities, and it investigates local temporal interactions across visual and audio, enabling the effective capture of cross-modal and cross-temporal dynamic events. Additionally, we present a cross-modal feature alignment module, which acts as a bridge to overcome the semantic gap among heterogeneous multi-modal data. It promotes the convergence of multimodal data distributions into a shared feature space, facilitating more effective and efficient processing. Extensive experimental results demonstrate the superiority of our method compared to state-of-the-art models across various question types on the challenging MUSIC-AVQA and AVQA benchmarks.

Graph-Based Multi-Interaction Network for Video Question Answering

Multi-interaction Network with Object Relation for Video Question Answering

Multi-Granularity Interaction and Integration Network for Video Question Answering

Cross-Attentional Spatio-Temporal Semantic Graph Networks for Video Question Answering

Multimodal Graph Reasoning and Fusion for Video Question Answering

Video Question Answering Via Multi-Granularity Temporal Attention Network Learning

Heterogeneous Interactive Graph Network for Audio-Visual Question Answering

Advancing Video Question Answering with a Multi-modal and Multi-layer Question Enhancement Network

Video Question Answering Via Grounded Cross-Attention Network Learning.

Multi-Turn Video Question Answering Via Multi-Stream Hierarchical Attention Context Network

Progressive Graph Attention Network for Video Question Answering

Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering

Multi-Turn Video Question Answering via Hierarchical Attention Context Reinforced Networks

Bottom-Up Hierarchical Propagation Networks with Heterogeneous Graph Modeling for Video Question Answering

Hierarchical Temporal Fusion of Multi-grained Attention Features for Video Question Answering

FHGN: Frame-Level Heterogeneous Graph Networks for Video Question Answering

Multi-Scale Progressive Attention Network for Video Question Answering

Multi-object event graph representation learning for Video Question Answering

Complementary Spatiotemporal Network for Video Question Answering.

Long-Term Video Question Answering Via Multimodal Hierarchical Memory Attentive Networks

Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering