Abstract:Visually-situated languages such as charts and plots are omnipresent in real-world documents. These graphical depictions are human-readable and are often analyzed in visually-rich documents to address a variety of questions that necessitate complex reasoning and common-sense responses. Despite the growing number of datasets that aim to answer questions over charts, most only address this task in isolation, without considering the broader context of document-level question answering. Moreover, such datasets lack adequate common-sense reasoning information in their questions. In this work, we introduce a novel task named document-level chart question answering (DCQA). The goal of this task is to conduct document-level question answering, extracting charts or plots in the document via document layout analysis (DLA) first and subsequently performing chart question answering (CQA). The newly developed benchmark dataset comprises 50,010 synthetic documents integrating charts in a wide range of styles (6 styles in contrast to 3 for PlotQA and ChartQA) and includes 699,051 questions that demand a high degree of reasoning ability and common-sense understanding. Besides, we present the development of a potent question-answer generation engine that employs table data, a rich color set, and basic question templates to produce a vast array of reasoning question-answer pairs automatically. Based on DCQA, we devise an OCR-free transformer for document-level chart-oriented understanding, capable of DLA and answering complex reasoning and common-sense questions over charts in an OCR-free manner. Our DCQA dataset is expected to foster research on understanding visualizations in documents, especially for scenarios that require complex reasoning for charts in the visually-rich document. We implement and evaluate a set of baselines, and our proposed method achieves comparable results.

DeMatch: Towards Understanding the Panel of Chart Documents

ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules

Towards an Efficient Framework for Data Extraction from Chart Images

Improving Machine Understanding of Human Intent in Charts

Advancing Chart Question Answering with Robust Chart Component Recognition

DCQA: Document-Level Chart Question Answering towards Complex Reasoning and Common-Sense Understanding

ChartBench: A Benchmark for Complex Visual Reasoning in Charts

Chart Decoder: Generating Textual and Numeric Information from Chart Images Automatically

From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models

Rethinking Comprehensive Benchmark for Chart Understanding: A Perspective from Scientific Literature

AskChart: Universal Chart Understanding through Textual Enhancement

StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding

MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning

ChartifyText: Automated Chart Generation from Data-Involved Texts via LLM

Enhancing Question Answering on Charts Through Effective Pre-training Tasks

A Hybrid Approach for Document Layout Analysis in Document images

OneChart: Purify the Chart Structural Extraction via One Auxiliary Token

The Story Behind the Lines: Line Charts as a Gateway to Dataset Discovery

ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization

Context-Aware Chart Element Detection

Mystique: Deconstructing SVG Charts for Layout Reuse