ChartKG: A Knowledge-Graph-Based Representation for Chart Images

Zhiguang Zhou,Haoxuan Wang,Zhengqing Zhao,Fengling Zheng,Yongheng Wang,Wei Chen,Yong Wang

2024-10-13

Abstract:Chart images, such as bar charts, pie charts, and line charts, are explosively produced due to the wide usage of data visualizations. Accordingly, knowledge mining from chart images is becoming increasingly important, which can benefit downstream tasks like chart retrieval and knowledge graph completion. However, existing methods for chart knowledge mining mainly focus on converting chart images into raw data and often ignore their visual encodings and semantic meanings, which can result in information loss for many downstream tasks. In this paper, we propose ChartKG, a novel knowledge graph (KG) based representation for chart images, which can model the visual elements in a chart image and semantic relations among them including visual encodings and visual insights in a unified manner. Further, we develop a general framework to convert chart images to the proposed KG-based representation. It integrates a series of image processing techniques to identify visual elements and relations, e.g., CNNs to classify charts, yolov5 and optical character recognition to parse charts, and rule-based methods to construct graphs. We present four cases to illustrate how our knowledge-graph-based representation can model the detailed visual elements and semantic relations in charts, and further demonstrate how our approach can benefit downstream applications such as semantic-aware chart retrieval and chart question answering. We also conduct quantitative evaluations to assess the two fundamental building blocks of our chart-to-KG framework, i.e., object recognition and optical character recognition. The results provide support for the usefulness and effectiveness of ChartKG.

Artificial Intelligence,Information Retrieval

What problem does this paper attempt to address?

The problem this paper attempts to address is: Existing chart knowledge extraction methods mainly focus on converting chart images into raw data, while neglecting the visual encoding and semantic information in charts, which can lead to information loss in many downstream tasks. Therefore, the authors propose a knowledge graph-based chart image representation method (ChartKG), aiming to represent the visual elements and their relationships in chart images in a unified and interpretable manner, thereby better supporting downstream tasks such as chart retrieval and chart question answering.

Specifically, the goals of the paper include:
1. **Propose a new knowledge graph-based chart image representation method**: This representation method can model the visual elements and their semantic relationships in charts in a unified and expressive manner, including visual encoding and visual insights.
2. **Design a general framework**: This framework can automatically convert chart images into the proposed knowledge graph representation method.
3. **Validate the effectiveness of the method through case studies and quantitative evaluations**: Demonstrate the application value of this method in downstream tasks such as chart retrieval and chart question answering.

Through these goals, the paper hopes to fill the gaps in existing methods in chart knowledge representation and provide a more comprehensive and effective solution.