Abstract:Visual question answering (VQA) is a multimodal task requiring a simultaneous understanding of both visual and textual content. Therefore, image and question comprehension, finding a dense interaction among words and regions, and inference knowledge are the cores of VQA. In this paper, we propose the Advanced Visual and Textual Co-context Aware Attention Network with Dependent Multimodal Fusion Block for Visual Question Answering (ACOCAD), consisting of the image and the question representations and three proposed mechanisms: textual context-aware attention, a question-level & word-level visual attention, and a dependent multimodal fusion block. The textual context-aware attention mechanism marks the keywords of the question and captures rich features by modeling a context-aware unit beside the Universal Sentence Encoder model (USE) and a self-attention unit. The advanced visual attention approach is applied to attend on the regions with the aim of question-level and word-level visual attention. The dependent multimodal fusion block is employed to enhance associating keywords with key regions and generate more efficient vectors. Three sub-models are defined based on the three proposed mechanisms, and one ablation study is conducted on the benchmarks GQA and VQA-v2 datasets to evaluate the effectiveness of each mechanism of our ACOCAD model. Then, another ablation study for the overall accuracy of the ACOCAD model is carried out on one of the hyper-parameters to find its optimal value. Moreover, we explore how the Dependent Multimodal Fusion Block may relieve limitations of prior methods in answering questions including homograph words. In addition, to address the challenge regarding the length of question words, the potential efficiency of the USE model and the Visual Attention Mechanism are analyzed. For further review, a qualitative evaluation is done to visualize the effectiveness of the ACOCAD model using some samples. The results demonstrate that the ACOCAD model outperforms four out of seven criteria in the GQA dataset, and its overall accuracy criterion reaches 57.37%. Furthermore, our model achieves a significant enhancement compared to the previous state-of-the-art models and reaches 87.43%, 71.02%, and 71.18% accuracies in the Yes/No question type, overall test-dev dataset, and overall test-std of VQA-v2, respectively. Moreover, one of these sub-models attains the best accuracy of 60.95% among all models for the other question type.

AMAM: An Attention-based Multimodal Alignment Model for Medical Visual Question Answering

MuVAM: A Multi-View Attention-based Model for Medical Visual Question Answering

Asymmetric cross-modal attention network with multimodal augmented mixup for medical visual question answering

Medical visual question answering with symmetric interaction attention and cross-modal gating

Medical visual question answering via corresponding feature fusion combined with semantic attention

VQAMix: Conditional Triplet Mixup for Medical Visual Question Answering

Cross-Modal Self-Supervised Vision Language Pre-training with Multiple Objectives for Medical Visual Question Answering

UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering

Multi-Modal Alignment of Visual Question Answering Based on Multi-Hop Attention Mechanism

Multi-level Relational Reasoning and Attention Network for Medical Visual Question Answering

Medical visual question answering based on question-type reasoning and semantic space constraint

Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answering

A Bi-level representation learning model for medical visual question answering

[A medical visual question answering approach based on co-attention networks]

Parallel multi-head attention and term-weighted question embedding for medical visual question answering

Caption-Aware Medical VQA Via Semantic Focusing and Progressive Cross-Modality Comprehension

Is Unimodal Bias Always Bad for Visual Question Answering? A Medical Domain Study with Dynamic Attention

Adversarial Learning with Multi-Modal Attention for Visual Question Answering.

Cross-Modal Self-Attention with Multi-Task Pre-Training for Medical Visual Question Answering

PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering

Advanced Visual and Textual Co-context Aware Attention Network with Dependent Multimodal Fusion Block for Visual Question Answering