Abstract:Recent advancements in Text-to-SQL have pushed database management systems towards greater democratization of data access. Today's language models are at the core of these advancements. They enable impressive Text-to-SQL generation as experienced in the development of Distyl AI's Analytics Insight Engine. Its early deployment with enterprise customers has highlighted three core challenges. First, data analysts expect support with authoring SQL queries of very high complexity. Second, requests are ad-hoc and, as such, require low latency. Finally, generation requires an understanding of domain-specific terminology and practices. The design and implementation of our Text-to-SQL generation pipeline, powered by large language models, tackles these challenges. The core tenants of our approach rely on external knowledge that we extract in a pre-processing phase, on retrieving the appropriate external knowledge at query generation time, and on decomposing SQL query generation following a hierarchical CTE-based structure. Finally, an adaptation framework leverages feedback to update the external knowledge, in turn improving query generation over time. We give an overview of our end-to-end approach and highlight the operators generating SQL during inference.

What problem does this paper attempt to address?

The paper primarily addresses several core challenges faced when utilizing large language models for end-to-end text-to-SQL generation in data analysis insight engines. Specifically, the research aims to: 1. **Support highly complex SQL query writing**: Data analysts expect the system to generate very complex SQL queries, which poses higher demands on existing natural language processing technologies. 2. **Low latency response**: Since query requests are often ad-hoc and random, the system needs to generate SQL queries in a very short time (up to 60 seconds, with an average of 30 seconds). 3. **Understanding domain-specific terms and practices**: To ensure that the generated queries correctly reflect the user's intent, the system also needs to have the ability to understand and handle domain-specific knowledge. To address the above challenges, the authors propose a text-to-SQL generation pipeline design and implementation method based on large language models. This method mainly includes three stages: preprocessing, inference, and adaptive learning. - **Preprocessing stage**: Construct an external knowledge set, including examples, guiding statements, and database schema representations extracted from historical query logs and domain documents. - **Inference stage**: Given a natural language input query, generate the SQL query through a series of operators, including retrieving relevant external knowledge, reconstructing the query, selecting guiding statements and examples, and generating the SQL query. - **Adaptive learning stage**: Update the external knowledge set based on user feedback and error information from query execution results to continuously improve the system's performance. The paper also details the specific implementation details and technical choices of each stage and demonstrates how their solution meets the needs for high complexity query generation, low latency response, and domain-specific knowledge understanding. Through this series of technical means, the research aims to improve the quality and efficiency of the text-to-SQL generation task.

End-to-end Text-to-SQL Generation within an Analytics Insight Engine

SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging

A Survey of Large Language Model-Based Generative AI for Text-to-SQL: Benchmarks, Applications, Use Cases, and Challenges

EzSQL: An SQL intermediate representation for improving SQL-to-text Generation

SPSQL: Step-by-step Parsing Based Framework for Text-to-SQL Generation

XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL

A Survey of Deep Learning Based Text-to-sql Generation

Deep Learning Driven Natural Languages Text to SQL Query Conversion: A Survey

Speech-to-SQL: Towards Speech-driven SQL Query Generation From Natural Language Question

CatSQL: Towards Real World Natural Language to SQL Applications.

SelECT-SQL: Self-correcting ensemble Chain-of-Thought for Text-to-SQL

Enhancing Text-to-SQL Translation for Financial System Design

Natural Language Query Engine for Relational Databases using Generative AI

E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL

KG-SQL: Hybrid Knowledge-Guided Semantic Understanding for Text-to-SQL

ER-SQL: Learning Enhanced Representation for Text-to-SQL Using Table Contents.

AI-Assisted SQL Authoring at Industry Scale

MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQL

SQLformer: Deep Auto-Regressive Query Graph Generation for Text-to-SQL Translation

M-SQL: Multi-Task Representation Learning for Single-Table Text2sql Generation

CHESS: Contextual Harnessing for Efficient SQL Synthesis