Abstract:Background: Visual question answering in medical domain (VQA-Med) exhibits great potential for enhancing confidence in diagnosing diseases and helping patients better understand their medical conditions. One of the challenges in VQA-Med is how to better understand and combine the semantic features of medical images (e.g., X-rays, Magnetic Resonance Imaging(MRI)) and answer the corresponding questions accurately in unlabeled medical datasets. Method: We propose a novel Bi-branched model based on Parallel networks and Image retrieval for Medical Visual Question Answering (BPI-MVQA). The first branch of BPI-MVQA is a transformer structure based on a parallel network to achieve complementary advantages in image sequence feature and spatial feature extraction, and multi-modal features are implicitly fused by using the multi-head self-attention mechanism. The second branch is retrieving the similarity of image features generated by the VGG16 network to obtain similar text descriptions as labels. Result: The BPI-MVQA model achieves state-of-the-art results on three VQA-Med datasets, and the main metric scores exceed the best results so far by 0.2[Formula: see text], 1.4[Formula: see text], and 1.1[Formula: see text]. Conclusion: The evaluation results support the effectiveness of the BPI-MVQA model in VQA-Med. The design of the bi-branch structure helps the model answer different types of visual questions. The parallel network allows for multi-angle image feature extraction, a unique feature extraction method that helps the model better understand the semantic information of the image and achieve greater accuracy in the multi-classification of VQA-Med. In addition, image retrieval helps the model answer irregular, open-ended type questions from the perspective of understanding the information provided by images. The comparison of our method with state-of-the-art methods on three datasets also shows that our method can bring substantial improvement to the VQA-Med system.

DIEM: Decomposition-Integration Enhancing Multimodal Insights

Simple and Effective Visual Question Answering in a Single Modality

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

Visual Question Decomposition on Multimodal Large Language Models

Visual Question Answering With Dense Inter- and Intra-Modality Interactions

Medical visual question answering with symmetric interaction attention and cross-modal gating

DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations

UIT-Saviors at MEDVQA-GI 2023: Improving Multimodal Learning with Image Enhancement for Gastrointestinal Visual Question Answering

Context-aware Multi-level Question Embedding Fusion for visual question answering

DisAVR: Disentangled Adaptive Visual Reasoning Network for Diagram Question Answering

MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationale

Multitask Learning for Visual Question Answering

AMAM: An Attention-based Multimodal Alignment Model for Medical Visual Question Answering

Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering

DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception

Detached and Interactive Multimodal Learning

Decouple Before Interact: Multi-Modal Prompt Learning for Continual Visual Question Answering

Tackling Vision Language Tasks Through Learning Inner Monologues

RSMoDM: Multimodal Momentum Distillation Model for Remote Sensing Visual Question Answering

BPI-MVQA: a bi-branch model for medical visual question answering

MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models