Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs

Jinyang Li,Binyuan Hui,Ge Qu,Jiaxi Yang,Binhua Li,Bowen Li,Bailin Wang,Bowen Qin,Rongyu Cao,Ruiying Geng,Nan Huo,Xuanhe Zhou,Chenhao Ma,Guoliang Li,Kevin C.C. Chang,Fei Huang,Reynold Cheng,Yongbin Li

2023-11-15

Abstract:Text-to-SQL parsing, which aims at converting natural language instructions into executable SQLs, has gained increasing attention in recent years. In particular, Codex and ChatGPT have shown impressive results in this task. However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on database schema with few rows of database contents leaving the gap between academic study and real-world applications. To mitigate this gap, we present Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks, containing 12,751 pairs of text-to-SQL data and 95 databases with a total size of 33.4 GB, spanning 37 professional domains. Our emphasis on database values highlights the new challenges of dirty database contents, external knowledge between NL questions and database contents, and SQL efficiency, particularly in the context of massive databases. To solve these problems, text-to-SQL models must feature database value comprehension in addition to semantic parsing. The experimental results demonstrate the significance of database values in generating accurate text-to-SQLs for big databases. Furthermore, even the most effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution accuracy, which is still far from the human result of 92.96%, proving that challenges still stand. Besides, we also provide an efficiency analysis to offer insights into generating text-to-efficient-SQLs that are beneficial to industries. We believe that BIRD will contribute to advancing real-world applications of text-to-SQL research. The leaderboard and source code are available: <a class="link-external link-https" href="https://bird-bench.github.io/" rel="external noopener nofollow">this https URL</a>.

Computation and Language

What problem does this paper attempt to address?

The paper attempts to address the challenges faced by current Text-to-SQL conversion models when dealing with large-scale and noisy real-world databases. Specifically:

1. **Limitations of existing benchmark datasets**: The widely used benchmark datasets (such as Spider and WikiSQL) mainly focus on database schemas while neglecting the importance of database values in real-world applications, leading to a gap between academic research and practical applications.

2. **Noisy data types in large-scale real-world databases**: The paper finds that existing state-of-the-art models still struggle to generalize well when faced with noisy data in large-scale real-world databases.

3. **Need for external knowledge reasoning**: As the scale of databases grows, there is a need to incorporate external knowledge for comprehensive understanding, which poses higher demands on the models.

4. **SQL execution efficiency**: Existing benchmarks do not consider the actual execution efficiency of SQL queries, which is crucial for practical applications in the industry.

To address these issues, the paper proposes a new benchmark dataset called BIRD (BIg bench for LaRge-Scale Database grounded in text-to-SQL), containing 12,751 text-to-SQL pairs and 95 databases with a total size of 33.4GB, covering 37 professional domains. Through BIRD, researchers aim to bridge the gap between experimental environments and real-world application scenarios, and to promote the development of more efficient and accurate Text-to-SQL models. Experimental results show that even the most advanced models (such as GPT-4) achieve an execution accuracy of only 54.89%, which is far below human performance (92.96%), indicating that there are still many challenges to overcome in this field.