Abstract:Exploratory data analysis (EDA), coupled with SQL, is essential for data analysts involved in data exploration and analysis. However, data analysts often encounter two primary challenges: (1) the need to craft SQL queries skillfully, and (2) the requirement to generate suitable visualization types that enhance the interpretation of query results. Due to its significance, substantial research efforts have been made to explore different approaches to address these challenges, including leveraging large language models (LLMs). However, existing methods fail to meet real-world data exploration requirements primarily due to (1) complex database schema; (2) unclear user intent; (3) limited cross-domain generalization capability; and (4) insufficient end-to-end text-to-visualization capability. This paper presents TiInsight, an automated SQL-based cross-domain exploratory data analysis system. First, we propose hierarchical data context (i.e., HDC), which leverages LLMs to summarize the contexts related to the database schema, which is crucial for open-world EDA systems to generalize across data domains. Second, the EDA system is divided into four components (i.e., stages): HDC generation, question clarification and decomposition, text-to-SQL generation (i.e., TiSQL), and data visualization (i.e., TiChart). Finally, we implemented an end-to-end EDA system with a user-friendly GUI interface in the production environment at PingCAP. We have also open-sourced all APIs of TiInsight to facilitate research within the EDA community. Through extensive evaluations by a real-world user study, we demonstrate that TiInsight offers remarkable performance compared to human experts. Specifically, TiSQL achieves an execution accuracy of 86.3% on the Spider dataset using GPT-4. It also demonstrates state-of-the-art performance on the Bird dataset.

Survey on Interactive Data Exploration

Editorial: TKDD Special Issue on Interactive Data Exploration and Analytics

Isphere

INODE: Building an End-to-End Data Exploration System in Practice [Extended Vision]

Efficient Query Construction For Large Scale Data

Visual Data Exploration and Its Applications

Exploratory Search on Big Data

IDEBench: A Benchmark for Interactive Data Exploration

Towards Automated Cross-domain Exploratory Data Analysis through Large Language Models

Research and Implementation of Document-Relational Data Query Execution Tech-nology

A Survey on Information Visualization: Recent Advances and Challenges

FlashView

CubeExplorer: online exploration of data cubes.

OIDM: Online Interactive Data Mining

DEEPEYE: an Automatic Big Data Visualization Framework

Data-Intensive Science and Engineering:Requirements and Challenges

Keyword-based Search and Exploration on Databases

Intelligent Data Analysis and Its Challenges in Big Data Environment

Iexplore: Accelerating Exploratory Data Analysis by Predicting User Intention.

Interactively Discovering and Ranking Desired Tuples by Data Exploration

Exploration and Visualization in the Web of Big Linked Data: A Survey of the State of the Art