Abstract:Real-world enterprise text-to-SQL workflows often involve complex cloud or local data across various database systems, multiple SQL queries in various dialects, and diverse operations from data transformation to analytics. We introduce Spider 2.0, an evaluation framework comprising 632 real-world text-to-SQL workflow problems derived from enterprise-level database use cases. The databases in Spider 2.0 are sourced from real data applications, often containing over 1,000 columns and stored in local or cloud database systems such as BigQuery and Snowflake. We show that solving problems in Spider 2.0 frequently requires understanding and searching through database metadata, dialect documentation, and even project-level codebases. This challenge calls for models to interact with complex SQL workflow environments, process extremely long contexts, perform intricate reasoning, and generate multiple SQL queries with diverse operations, often exceeding 100 lines, which goes far beyond traditional text-to-SQL challenges. Our evaluations indicate that based on o1-preview, our code agent framework successfully solves only 17.0% of the tasks, compared with 91.2% on Spider 1.0 and 73.0% on BIRD. Our results on Spider 2.0 show that while language models have demonstrated remarkable performance in code generation -- especially in prior text-to-SQL benchmarks -- they require significant improvement in order to achieve adequate performance for real-world enterprise usage. Progress on Spider 2.0 represents crucial steps towards developing intelligent, autonomous, code agents for real-world enterprise settings. Our code, baseline models, and data are available at <a class="link-external link-https" href="https://spider2-sql.github.io" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### Problems the paper attempts to solve The paper aims to address the insufficiency of existing language models in handling enterprise - level text - to - SQL workflow tasks. Specifically: 1. **Complex enterprise - level database environment**: - Existing text - to - SQL benchmark tests usually use non - industrial - level databases. These databases have few tables and columns, and the SQL queries are simple and lack diversity. However, actual enterprise - level database systems contain thousands of columns, distributed in various local and cloud databases such as BigQuery and Snowflake. - SQL queries in practical applications often need to handle complex database metadata, multiple SQL dialect documents, and project - level code bases, which far exceed the complexity of traditional text - to - SQL tasks. 2. **Multi - step complex workflows**: - Enterprise - level text - to - SQL workflows not only require generating a single SQL query but also need to handle multiple SQL queries, perform complex data transformation and analysis operations, which may involve hundreds of lines of code. - The model needs to have dynamic interaction capabilities and be able to adjust the generated SQL queries according to the context and feedback to complete complex tasks. 3. **Diverse SQL dialects and advanced functions**: - SQL queries in practical applications often use dialects and advanced functions specific to a database. These functions may have different syntax and functions in different database systems. - The model needs to be able to understand and use these dialects and functions to generate correct SQL queries. 4. **Project - level code bases and external knowledge**: - Enterprise - level tasks usually require access to project - level code bases and external documents. These resources provide important context information to help the model generate more accurate SQL queries. - The model needs to be able to effectively utilize these resources, understand the project structure and database schema, to complete complex tasks. ### Solutions To meet the above challenges, the paper introduces Spider 2.0, an evaluation framework that contains 632 real - world enterprise - level text - to - SQL workflow problems. Spider 2.0 has the following features: - **Real databases and SQL queries**: The databases are sourced from practical applications, containing large - scale schemas and complex structures. Each SQL query contains an average of 144 tokens and 7.1 special functions on average. - **Diverse database systems**: It covers multiple database systems, including cloud data warehouses (such as BigQuery and Snowflake) and local databases (such as SQLite and DuckDB). - **Complex task settings**: The tasks not only require generating SQL queries but also require the model to be able to interact dynamically with the database and project code base and handle multi - step workflows. - **Detailed annotation process**: Through a six - step annotation process, the authenticity and complexity of the data are ensured, including database and SQL collection, SQL rewriting, code base and context setting, natural language task instruction annotation, execution - based focused evaluation, and quality control. ### Evaluation results - **Performance of existing models**: Even the most advanced models (such as o1 - preview) have a success rate of only 17.01% on Spider 2.0, far lower than their performance on traditional benchmark tests (such as Spider 1.0 and BIRD). - **Main challenges**: Models have significant difficulties in handling large databases, different SQL dialects, multi - step complex workflows, and project - level code bases. Through Spider 2.0, researchers hope to promote the development of a new generation of intelligent autonomous agents that can efficiently execute data engineering workflows in real - enterprise environments.

Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?

CatSQL: Towards Real World Natural Language to SQL Applications.

CodeS: Towards Building Open-source Language Models for Text-to-SQL

Tool-Assisted Agent on SQL Inspection and Refinement in Real-World Scenarios

Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

MultiSpider: Towards Benchmarking Multilingual Text-to-SQL Semantic Parsing

XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL

MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQL

Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation

Evaluating LLMs for Text-to-SQL Generation With Complex SQL Workload

Schema-Aware Multi-Task Learning for Complex Text-to-SQL

SA-SQL: A Schema-Aligned Framework for Text-to-SQL Through Large Language Models

Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs

Reboost Large Language Model-based Text-to-SQL, Text-to-Python, and Text-to-Function -- with Real Applications in Traffic Domain

SEA-SQL: Semantic-Enhanced Text-to-SQL with Adaptive Refinement

ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems

Analyzing the Effectiveness of Large Language Models on Text-to-SQL Synthesis

RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL

Ar-Spider: Text-to-SQL in Arabic

S$^2$SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers