Abstract:Text-to-SQL systems (also known as NL-to-SQL systems) have become an increasingly popular solution for bridging the gap between user capabilities and SQL-based data access. These systems translate user requests in natural language to valid SQL statements for a specific database. Recent Text-to-SQL systems have benefited from the rapid improvement of transformer-based language models. However, while Text-to-SQL systems that incorporate such models continuously reach new high scores on -- often synthetic -- benchmark datasets, a systematic exploration of their robustness towards different data models in a real-world, realistic scenario is notably missing. This paper provides the first in-depth evaluation of the data model robustness of Text-to-SQL systems in practice based on a multi-year international project focused on Text-to-SQL interfaces. Our evaluation is based on a real-world deployment of FootballDB, a system that was deployed over a 9 month period in the context of the FIFA World Cup 2022, during which about 6K natural language questions were asked and executed. All of our data is based on real user questions that were asked live to the system. We manually labeled and translated a subset of these questions for three different data models. For each data model, we explore the performance of representative Text-to-SQL systems and language models. We further quantify the impact of training data size, pre-, and post-processing steps as well as language model inference time. Our comprehensive evaluation sheds light on the design choices of real-world Text-to-SQL systems and their impact on moving from research prototypes to real deployments. Last, we provide a new benchmark dataset to the community, which is the first to enable the evaluation of different data models for the same dataset and is substantially more challenging than most previous datasets in terms of query complexity.

Bridging the gap between text-to-SQL research and real-world applications: A unified all-in-one framework for text-to-SQL

SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging

Improving Text-to-SQL with a Hybrid Decoding Method

SA-SQL: A Schema-Aligned Framework for Text-to-SQL Through Large Language Models

A Heterogeneous Graph to Abstract Syntax Tree Framework for Text-to-SQL

Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing

TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring

SPSQL: Step-by-step Parsing Based Framework for Text-to-SQL Generation

Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect

Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries

UNITE: A Unified Benchmark for Text-to-SQL Evaluation

On the Structural Generalization in Text-to-SQL

FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis

RHB-Net: A Relation-aware Historical Bridging Network for Text2SQL Auto-Completion

A BERT-Based Generation Model to Transform Medical Texts to SQL Queries for Electronic Medical Records: Model Development and Validation

SQLfuse: Enhancing Text-to-SQL Performance through Comprehensive LLM Synergy

XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL

Enhancing Text-to-SQL Translation for Financial System Design

Decoupling SQL Query Hardness Parsing for Text-to-SQL

Retrieval-augmented GPT-3.5-based Text-to-SQL Framework with Sample-aware Prompting and Dynamic Revision Chain

KU-DMIS at EHRSQL 2024:Generating SQL query via question templatization in EHR