Abstract:Text-Centric Visual Question Answering (TEC-VQA) in its proper format not only facilitates human-machine interaction in text-centric visual environments but also serves as a de facto gold proxy to evaluate AI models in the domain of text-centric scene understanding. Nonetheless, most existing TEC-VQA benchmarks have focused on high-resource languages like English and Chinese. Despite pioneering works to expand multilingual QA pairs in non-text-centric VQA datasets through translation engines, the translation-based protocol encounters a substantial "visual-textual misalignment" problem when applied to TEC-VQA. Specifically, it prioritizes the text in question-answer pairs while disregarding the visual text present in images. Moreover, it fails to address complexities related to nuanced meaning, contextual distortion, language bias, and question-type diversity. In this work, we tackle multilingual TEC-VQA by introducing MTVQA, the first benchmark featuring high-quality human expert annotations across 9 diverse languages, consisting of 6,778 question-answer pairs across 2,116 images. Further, by comprehensively evaluating numerous state-of-the-art Multimodal Large Language Models (MLLMs), including GPT-4o, GPT-4V, Claude3, and Gemini, on the MTVQA dataset, it is evident that there is still a large room for performance improvement, underscoring the value of MTVQA. Additionally, we supply multilingual training data within the MTVQA dataset, demonstrating that straightforward fine-tuning with this data can substantially enhance multilingual TEC-VQA performance. We aspire that MTVQA will offer the research community fresh insights and stimulate further exploration in multilingual visual text comprehension. The project homepage is available at <a class="link-external link-https" href="https://bytedance.github.io/MTVQA/" rel="external noopener nofollow">this https URL</a>.

Champion Solution for the WSDM2023 Toloka VQA Challenge

Second Place Solution of WSDM2023 Toloka Visual Question Answering Challenge

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

First Place Solution to the Multiple-choice Video QA Track of The Second Perception Test Challenge

The Solution for the ICCV 2023 Perception Test Challenge 2023 -- Task 6 -- Grounded videoQA

First Place Solution to the CVPR'2023 AQTC Challenge: A Function-Interaction Centric Approach with Spatiotemporal Visual-Language Alignment

MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering

Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQA

1st Place Solution for PVUW Challenge 2023: Video Panoptic Segmentation

VTQA2023: ACM Multimedia 2023 Visual Text Question Answering Challenge

VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning

Visual-Textual Semantic Alignment Network for Visual Question Answering

Finetuning Language Models for Multimodal Question Answering

STVGFormer

VimTS: A Unified Video and Image Text Spotter for Enhancing the Cross-domain Generalization

Unified Transformer with Cross-Modal Mixture Experts for Remote-Sensing Visual Question Answering

A Solution to CVPR'2023 AQTC Challenge: Video Alignment for Multi-Step Inference

Achieving Human Parity on Visual Question Answering

1st Place Winner of the 2024 Pixel-level Video Understanding in the Wild (CVPR'24 PVUW) Challenge in Video Panoptic Segmentation and Best Long Video Consistency of Video Semantic Segmentation

Positional Attention Guided Transformer-Like Architecture for Visual Question Answering

STVGFormer: Spatio-Temporal Video Grounding with Static-Dynamic Cross-Modal Understanding