Abstract:Chart understanding tasks such as ChartQA and Chart-to-Text involve automatically extracting and interpreting key information from charts, enabling users to query or convert visual data into structured formats. State-of-the-art approaches primarily focus on visual cues from chart images, failing to explicitly incorporate rich textual information (e.g., data labels and axis labels) embedded within the charts. This textual information is vital for intuitive human comprehension and interpretation of charts. Moreover, existing models are often large and computationally intensive, limiting their practical applicability. In this paper, we introduce AskChart, a universal model that explicitly integrates both textual and visual cues from charts using a Mixture of Experts (MoE) architecture. AskChart facilitates the learning of enhanced visual-textual representations of charts for effectively handling multiple chart understanding tasks, while maintaining a smaller model size. To capture the synergy between visual and textual modalities, we curate a large-scale dataset named ChartBank with about 7.5M data samples, which helps align textual and visual information and facilitates the extraction of visual entities and text. To effectively train AskChart, we design a three-stage training strategy to align visual and textual modalities for learning robust visual-textual representations and optimizing the learning of the MoE layer. Extensive experiments across five datasets demonstrate the significant performance gains of AskChart in four chart understanding tasks. Remarkably, AskChart with 4.6B parameters outperforms state-of-the-art models with 13B parameters by 68.3% in Open-ended ChartQA and 49.2% in Chart-to-Text tasks, while achieving comparable performance in ChartQA and Chart-to-Table tasks.

Chart2Vec: A Universal Embedding of Context-Aware Visualizations

Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative Model

An Intelligent Approach to Automatically Discovering Visual Insights

AskChart: Universal Chart Understanding through Textual Enhancement

Chartem: Reviving Chart Images with Data Embedding

MSG-Chart: Multimodal Scene Graph for ChartQA

A Visual Analysis Approach for Community Detection of Multi-Context Mobile Social Networks

Visualization Assessment - A Machine Learning Approach.

mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning

AdaVis: Adaptive and Explainable Visualization Recommendation for Tabular Data

ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization

EmbeddingVis: A Visual Analytics Approach to Comparative Network Embedding Inspection

Probing Contextualized Sentence Representations with Visual Awareness

CoVis: A Collaborative Framework for Fine-grained Graphic Visual Understanding

ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning

Beyond Embeddings: The Promise of Visual Table in Visual Reasoning

Is Visual Context Really Helpful for Knowledge Graph? A Representation Learning Perspective.

AutoCaption: an Approach to Generate Natural Language Description from Visualization Automatically

InvVis: Large-Scale Data Embedding for Invertible Visualization