Abstract:Audio-visual question answering (AVQA) is an emerging task that aims to provide answers by integrating visual contents, audio streams, and their associations within given videos. The major challenge lies in effectively fusing heterogeneous multi-modal data to comprehend complex scenes while capturing question-related clues to infer correct answers. Current AVQA models primarily employ attention mechanisms to extract questionrelated clues separately from visual and audio modalities before combining them. However, these approaches have two limitations: (1) They neglect the exploration of the association and complementary between audio and visual; (2) Encoding visual or audio holistically limits the capacity to capture the cross-modal and crosstemporal dynamic events. In this paper, we introduce the Heterogeneous Interactive Graph Network, a novel solution designed to address these limitations. Specifically, we construct heterogeneous multi-modal graphs that facilitate unified integration of multiple modalities, including visual, audio, and question. This approach effectively explores the associations and complementarity among multiple modalities, and it investigates local temporal interactions across visual and audio, enabling the effective capture of cross-modal and cross-temporal dynamic events. Additionally, we present a cross-modal feature alignment module, which acts as a bridge to overcome the semantic gap among heterogeneous multi-modal data. It promotes the convergence of multimodal data distributions into a shared feature space, facilitating more effective and efficient processing. Extensive experimental results demonstrate the superiority of our method compared to state-of-the-art models across various question types on the challenging MUSIC-AVQA and AVQA benchmarks.

Multi-interaction Network with Object Relation for Video Question Answering

Graph-Based Multi-Interaction Network for Video Question Answering

Multi-Granularity Interaction and Integration Network for Video Question Answering

Video Question Answering Via Grounded Cross-Attention Network Learning.

Video Question Answering Via Multi-Granularity Temporal Attention Network Learning

Modular Blended Attention Network for Video Question Answering

Multichannel Attention Refinement for Video Question Answering.

Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering

Frame Augmented Alternating Attention Network for Video Question Answering.

Multi-Turn Video Question Answering Via Multi-Stream Hierarchical Attention Context Network

Multi-Turn Video Question Answering via Hierarchical Attention Context Reinforced Networks

Video Question Answering via Attribute-Augmented Attention Network Learning

Multi-object event graph representation learning for Video Question Answering

Multimodal Graph Reasoning and Fusion for Video Question Answering

Temporal Pyramid Transformer with Multimodal Interaction for Video Question Answering

Video Question Answering via Knowledge-based Progressive Spatial-Temporal Attention Network

Heterogeneous Interactive Graph Network for Audio-Visual Question Answering

Hierarchical Temporal Fusion of Multi-grained Attention Features for Video Question Answering

Video Question Answering Via Hierarchical Dual-Level Attention Network Learning.

Multi-Granularity Relational Attention Network for Audio-Visual Question Answering

Adversarial Multimodal Network for Movie Question Answering