Abstract:Visual Question Answering (VQA) has recently emerged as a potential research domain, captivating the interest of many in the field of artificial intelligence and computer vision. Despite the prevalence of approaches in English, there is a notable lack of systems specifically developed for certain languages, particularly Vietnamese. This study aims to bridge this gap by conducting comprehensive experiments on the Vietnamese Visual Question Answering (ViVQA) dataset, demonstrating the effectiveness of our proposed model. In response to community interest, we have developed a model that enhances image representation capabilities, thereby improving overall performance in the ViVQA system. Specifically, our model integrates the Bootstrapping Language-Image Pre-training with frozen unimodal models (BLIP-2) and the convolutional neural network EfficientNet to extract and process both local and global features from images. This integration leverages the strengths of transformer-based architectures for capturing comprehensive contextual information and convolutional networks for detailed local features. By freezing the parameters of these pre-trained models, we significantly reduce the computational cost and training time, while maintaining high performance. This approach significantly improves image representation and enhances the performance of existing VQA systems. We then leverage a multi-modal fusion module based on a general-purpose multi-modal foundation model (BEiT-3) to fuse the information between visual and textual features. Our experimental findings demonstrate that our model surpasses competing baselines, achieving promising performance. This is particularly evident in its accuracy of $71.04\%$ on the test set of the ViVQA dataset, marking a significant advancement in our research area. The code is available at <a class="link-external link-https" href="https://github.com/nngocson2002/ViVQA" rel="external noopener nofollow">this https URL</a>.

CLVIN: Complete language-vision interaction network for visual question answering

Simple and Effective Visual Question Answering in a Single Modality

Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integration

Decouple Before Interact: Multi-Modal Prompt Learning for Continual Visual Question Answering

Dual-decoder transformer network for answer grounding in visual question answering

A lightweight Transformer-based visual question answering network with Weight-Sharing Hybrid Attention

Dual-feature collaborative relation-attention networks for visual question answering

Deep Attention Neural Tensor Network For Visual Question Answering

LCV2: A Universal Pretraining-Free Framework for Grounded Visual Question Answering

Deep Modular Co-Attention Networks for Visual Question Answering

Cross-Modal Multistep Fusion Network with Co-Attention for Visual Question Answering

Positional Attention Guided Transformer-Like Architecture for Visual Question Answering

Visual Question Answering With Dense Inter- and Intra-Modality Interactions

LCV2: An Efficient Pretraining-Free Framework for Grounded Visual Question Answering

VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers

BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions

Hierarchical Vision and Language Transformer for Efficient Visual Dialog

Co-attention graph convolutional network for visual question answering

Towards Better Vision-Inspired Vision-Language Models