ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization

Mengsha Liu,Daoyuan Chen,Yaliang Li,Guian Fang,Ying Shen

2024-04-25

Abstract:Data visualization serves as a critical means for presenting data and mining its valuable insights. The task of chart summarization, through natural language processing techniques, facilitates in-depth data analysis of charts. However, there still are notable deficiencies in terms of visual-language matching and reasoning ability for existing approaches. To address these limitations, this study constructs a large-scale dataset of comprehensive chart-caption pairs and fine-tuning instructions on each chart. Thanks to the broad coverage of various topics and visual styles within this dataset, better matching degree can be achieved from the view of training data. Moreover, we propose an innovative chart summarization method, ChartThinker, which synthesizes deep analysis based on chains of thought and strategies of context retrieval, aiming to improve the logical coherence and accuracy of the generated summaries. Built upon the curated datasets, our trained model consistently exhibits superior performance in chart summarization tasks, surpassing 8 state-of-the-art models over 7 evaluation metrics. Our dataset and codes are publicly accessible.

Computation and Language

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper "ChartThinker: An Optimized Chart Summarization Method Based on Contextual Chain of Thought" aims to address the deficiencies in existing chart summarization methods in terms of visual-language matching and reasoning capabilities. Specifically, the authors identify the following two main issues:

1. **Insufficient Matching**:
- **Incomplete Descriptions**: Due to the small proportion of numbers and text in charts, existing models tend to overlook these details, leading to content omissions.
- **Fabricated Chart Content**: Large visual-language models may generate text influenced by irrelevant information from the pre-training corpus, resulting in content unrelated to the chart.

2. **Reasoning Errors**:
- Large models perform poorly in chart reasoning tasks. Besides numerical descriptions, the model needs to provide a comprehensive summary that captures the chart's intent. However, some charts contain a large amount of numerical data and complex curve patterns, making it difficult for the model to accurately understand the chart's intrinsic meaning, leading to reasoning errors.

To address these issues, the authors propose a new method—ChartThinker, which combines Chain of Thought (CoT) and contextual retrieval to generate textual descriptions of charts. This method aims to improve the logical coherence and accuracy of the generated summaries.