Abstract:Video Question Answering (VideoQA) requires a thorough comprehension of linguistic and visual modalities. However, recent methods confront two problems: (1) Synchronous modeling of object action and frame scene instead of a step-by-step manner, which can better mine potential semantic attributes of videos, lacks research; (2) The relationship between cross-modal alignments at different granularity of abstraction is not fully utilized. Based on these insights, we propose a novel method named hierarchical synchronization with structured multi-granularity interaction (HSSMI) for VideoQA. First, a hierarchical synchronous reasoning module is put forward to model objects’ relations and dynamics and synchronously capture their synergistic influences over time when analyzing whole frames. It is seen as multiple Object ConvLSTMs (O-CLSTMs) in isolation or a Frame ConvLSTM (F-CLSTM) in collectivity. Specifically, O-CLSTM learns the object-level action states under neighboring spatial interplays. Meanwhile, F-CLSTM learns the frame-level scene state, where action information from O-CLSTMs is selectively aggregated into a common memory cell of F-CLSTM as instructed by questions. Besides, a boundary detector is equipped to discover scene discontinuities, enabling F-CLSTM to alter its time connectivity and adapt its sequential encoding process to videos. Thereafter, we develop a conditional VLAD with topic constraints for discriminative modality summarization. Last, a structured multi-granularity interaction module is proposed to integrate complemented clues on the global alignment between scene and full question and the local alignments between action summaries and words. This module encourages useful information passing through compositional syntactical topologies of questions to predict answers. Experiments on three public benchmark datasets demonstrate the superiority of our HSSMI against other state-of-the-art methods. Codes will be publicly available at https://github.com/Qiss33/HSSMI.

Fine-Grained Question-Answer Matching via Sentence-Aware Contrastive Self-supervised Transfer.

S2M: Converting Single-Turn to Multi-Turn Datasets for Conversational Question Answering

Heterogeneous Community Question Answering via Social-Aware Multi-Modal Co-Attention Convolutional Matching

CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert Judgments For Open-Domain Question Answering

QAVidCap: Enhancing Video Captioning Through Question Answering Techniques

Multi-view Semantic Matching of Question retrieval using Fine-grained Semantic Representations

Multi-Scale Fine-Grained Alignments for Image and Sentence Matching

One Stone, Four Birds: A Comprehensive Solution for QA System Using Supervised Contrastive Learning

Supervised and Unsupervised Transfer Learning for Question Answering

IARNN-Based Semantic-Containing Double-Level Embedding Bi-LSTM for Question-and-Answer Matching

Frame-Subtitle Self-Supervision for Multi-Modal Video Question Answering

FiTs: Fine-grained Two-stage Training for Knowledge-aware Question Answering.

Long Text QA Matching Based on ESIM of Fusion Convolution Feature

BERT-Based Mixed Question Answering Matching Model

Chinese Semantic Matching with Multi-granularity Alignment and Feature Fusion

Exploiting Sentence Embedding for Medical Question Answering

Enhanced Answer Selection in CQA Using Multi-Dimensional Features Combination

Towards Fine-grained Text Sentiment Transfer

Cross-Modal Semantic Alignment before Fusion for Two-Pass End-to-End Spoken Language Understanding

Hierarchical synchronization with structured multi-granularity interaction for video question answering