Abstract:Video Question Answering (Video QA) challenges modelers in multiple fronts. Modeling video necessitates building not only spatio-temporal models for the dynamic visual channel but also multimodal structures for associated information channels such as subtitles or audio. Video QA adds at least two more layers of complexity – selecting relevant content for each channel in the context of the linguistic query, and composing spatio-temporal concepts and relations hidden in the data in response to the query. To address these requirements, we start with two insights: (a) content selection and relation construction can be jointly encapsulated into a conditional computational structure, and (b) video-length structures can be composed hierarchically. For (a) this paper introduces a general-reusable reusable neural unit dubbed Conditional Relation Network (CRN) taking as input a set of tensorial objects and translating into a new set of objects that encode relations of the inputs. The generic design of CRN helps ease the common complex model building process of Video QA by simple block stacking and rearrangements with flexibility in accommodating diverse input modalities and conditioning features across both visual and linguistic domains. As a result, we realize insight (b) by introducing Hierarchical Conditional Relation Networks (HCRN) for Video QA. The HCRN primarily aims at exploiting intrinsic properties of the visual content of a video as well as its accompanying channels in terms of compositionality, hierarchy, and near-term and far-term relation. HCRN is then applied for Video QA in two forms, short-form where answers are reasoned solely from the visual content of a video, and long-form where an additional associated information channel, such as movie subtitles, presented. Our rigorous evaluations show consistent improvements over state-of-the-art methods on well-studied benchmarks including large-scale real-world datasets such as TGIF-QA and TVQA, demonstrating the strong capabilities of our CRN unit and the HCRN for complex domains such as Video QA. To the best of our knowledge, the HCRN is the very first method attempting to handle long and short-form multimodal Video QA at the same time.

Hierarchical synchronization with structured multi-granularity interaction for video question answering

Multi-Granularity Semantic Clues Extraction for Video Question Answering

Multi-Granularity Interaction and Integration Network for Video Question Answering

Hierarchical Temporal Fusion of Multi-grained Attention Features for Video Question Answering

Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Long-Term Video Question Answering Via Multimodal Hierarchical Memory Attentive Networks

Video Question Answering with Semantic Disentanglement and Reasoning

Multi-Turn Video Question Answering Via Multi-Stream Hierarchical Attention Context Network

A multi-scale self-supervised hypergraph contrastive learning framework for video question answering

Fusing Temporally Distributed Multi-Modal Semantic Clues for Video Question Answering.

Bottom-Up Hierarchical Propagation Networks with Heterogeneous Graph Modeling for Video Question Answering

Complementary Spatiotemporal Network for Video Question Answering.

Relation-aware Hierarchical Attention Framework for Video Question Answering

Heterogeneous Interactive Graph Network for Audio-Visual Question Answering

Hierarchical Multi-Modality Graph Reasoning for Remote Sensing Visual Question Answering

Cross-Attentional Spatio-Temporal Semantic Graph Networks for Video Question Answering

SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answering

Hierarchical Conditional Relation Networks for Multimodal Video Question Answering

Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

HSM-QA: Question Answering System Based on Hierarchical Semantic Matching