Abstract:In this paper, we propose a novel multi-modal framework for Scene Text Visual Question Answering (STVQA), which requires models to read scene text in images for question answering. Apart from text or visual objects, which could exist independently, scene text naturally links text and visual modalities together by conveying linguistic semantics while being a visual object in an image simultaneously. Different to conventional STVQA models which take the linguistic semantics and visual semantics in scene text as two separate features, in this paper, we propose a paradigm of "Locate Then Generate" (LTG), which explicitly unifies this two semantics with the spatial bounding box as a bridge connecting them. Specifically, at first, LTG locates the region in an image that may contain the answer words with an answer location module (ALM) consisting of a region proposal network and a language refinement network, both of which can transform to each other with one-to-one mapping via the scene text bounding box. Next, given the answer words selected by ALM, LTG generates a readable answer sequence with an answer generation module (AGM) based on a pre-trained language model. As a benefit of the explicit alignment of the visual and linguistic semantics, even without any scene text based pre-training tasks, LTG can boost the absolute accuracy by +6.06% and +6.92% on the TextVQA dataset and the ST-VQA dataset respectively, compared with a non-pre-training baseline. We further demonstrate that LTG effectively unifies visual and text modalities through the spatial bounding box connection, which is underappreciated in previous methods.

VTQAGen: BART-based Generative Model For Visual Text Question Answering

Dual Path Multi-Modal High-Order Features for Textual Content Based Visual Question Answering

Simple and Effective Visual Question Answering in a Single Modality

VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning

TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation

Multimodal Bi-direction Guided Attention Networks for Visual Question Answering

Question-relationship guided graph attention network for visual question answer

Visual Question Answering As Reading Comprehension

See and Learn More: Dense Caption-Aware Representation for Visual Question Answering

VTQA2023: ACM Multimedia 2023 Visual Text Question Answering Challenge

Finetuning Language Models for Multimodal Question Answering

AI-VQA

Multi-modal Contextual Graph Neural Network for Text Visual Question Answering.

AI-VQA: Visual Question Answering based on Agent Interaction with Interpretability

Exploring Diverse Methods in Visual Question Answering

Visual question answering algorithm based on image caption

Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQA

All You May Need for VQA are Image Captions

The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions

Multiple Objects-Aware Visual Question Generation