Abstract:Video question answering (VideoQA) is a challenging yet important task that requires a joint understanding of low-level video content and high-level textual semantics. Despite the promising progress of existing efforts, recent studies revealed that current VideoQA models mostly tend to over-rely on the superficial correlations rooted in the dataset bias while overlooking the key video content, thus leading to unreliable results. Effectively understanding and modeling the temporal and semantic characteristics of a given video for robust VideoQA is crucial but, to our knowledge, has not been well investigated. To fill the research gap, we propose a robust VideoQA framework that can effectively model the cross-modality fusion and enforce the model to focus on the temporal and global content of videos when making a QA decision instead of exploiting the shortcuts in datasets. Specifically, we design a self-supervised contrastive learning objective to contrast the positive and negative pairs of multimodal input, where the fused representation of the original multimodal input is enforced to be closer to that of the intervened input based on video perturbation. We expect the fused representation to focus more on the global context of videos rather than some static keyframes. Moreover, we introduce an effective temporal order regularization to enforce the inherent sequential structure of videos for video representation. We also design a Kullback-Leibler divergence-based perturbation invariance regularization of the predicted answer distribution to improve the robustness of the model against temporal content perturbation of videos. Our method is model-agnostic and can be easily compatible with various VideoQA backbones. Extensive experimental results and analyses on several public datasets show the advantage of our method over the state-of-the-art methods in terms of both accuracy and robustness.

Fusing Temporally Distributed Multi-Modal Semantic Clues for Video Question Answering.

Video Question Answering with Semantic Disentanglement and Reasoning

Cross-Attentional Spatio-Temporal Semantic Graph Networks for Video Question Answering

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Hierarchical Temporal Fusion of Multi-grained Attention Features for Video Question Answering

Multichannel Attention Refinement for Video Question Answering.

Multi-Granularity Semantic Clues Extraction for Video Question Answering

Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

Advancing Video Question Answering with a Multi-modal and Multi-layer Question Enhancement Network

Multimodal Graph Reasoning and Fusion for Video Question Answering

Video question answering via gradually refined attention over appearance and motion

Complementary Spatiotemporal Network for Video Question Answering.

Modality attention fusion model with hybrid multi-head self-attention for video understanding

Long-Term Video Question Answering Via Multimodal Hierarchical Memory Attentive Networks

Robust video question answering via contrastive cross-modality representation learning

Multimodal Dual Attention Memory for Video Story Question Answering

Video Question Answering Via Multi-Granularity Temporal Attention Network Learning

Instance-sequence reasoning for video question answering

Cross-Modal Multistep Fusion Network with Co-Attention for Visual Question Answering

Temporal Pyramid Transformer with Multimodal Interaction for Video Question Answering