Abstract:Text-Centric Visual Question Answering (TEC-VQA) in its proper format not only facilitates human-machine interaction in text-centric visual environments but also serves as a de facto gold proxy to evaluate AI models in the domain of text-centric scene understanding. Nonetheless, most existing TEC-VQA benchmarks have focused on high-resource languages like English and Chinese. Despite pioneering works to expand multilingual QA pairs in non-text-centric VQA datasets through translation engines, the translation-based protocol encounters a substantial "visual-textual misalignment" problem when applied to TEC-VQA. Specifically, it prioritizes the text in question-answer pairs while disregarding the visual text present in images. Moreover, it fails to address complexities related to nuanced meaning, contextual distortion, language bias, and question-type diversity. In this work, we tackle multilingual TEC-VQA by introducing MTVQA, the first benchmark featuring high-quality human expert annotations across 9 diverse languages, consisting of 6,778 question-answer pairs across 2,116 images. Further, by comprehensively evaluating numerous state-of-the-art Multimodal Large Language Models (MLLMs), including GPT-4o, GPT-4V, Claude3, and Gemini, on the MTVQA dataset, it is evident that there is still a large room for performance improvement, underscoring the value of MTVQA. Additionally, we supply multilingual training data within the MTVQA dataset, demonstrating that straightforward fine-tuning with this data can substantially enhance multilingual TEC-VQA performance. We aspire that MTVQA will offer the research community fresh insights and stimulate further exploration in multilingual visual text comprehension. The project homepage is available at <a class="link-external link-https" href="https://bytedance.github.io/MTVQA/" rel="external noopener nofollow">this https URL</a>.

CFMMC-Align: Coarse-Fine Multi-Modal Contrastive Alignment Network for Traffic Event Video Question Answering

Video Question Answering Via Grounded Cross-Attention Network Learning.

Video question answering via traffic knowledge database and question classification

Frame Augmented Alternating Attention Network for Video Question Answering.

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Eyes on the Road: State-of-the-Art Video Question Answering Models Assessment for Traffic Monitoring Tasks

Fine-grained Cross-modal Alignment Network for Text-Video Retrieval

Hierarchical Temporal Fusion of Multi-grained Attention Features for Video Question Answering

Video Question Answering Via Multi-Granularity Temporal Attention Network Learning

Progressive Graph Attention Network for Video Question Answering

Cross-Modal Multistep Fusion Network with Co-Attention for Visual Question Answering

TASTA: Text-Assisted Spatial and Temporal Attention Network for Video Question Answering.

Cross-Attentional Spatio-Temporal Semantic Graph Networks for Video Question Answering

Long-Term Video Question Answering Via Multimodal Hierarchical Memory Attentive Networks

Visual-Textual Semantic Alignment Network for Visual Question Answering

TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioning

Contrastive Fusion Representation: Mitigating Adversarial Attacks on VQA Models

Multichannel Attention Refinement for Video Question Answering.

MFVC: Urban Traffic Scene Video Caption Based on Multimodal Fusion

MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering

Advancing Video Question Answering with a Multi-modal and Multi-layer Question Enhancement Network