Abstract:Equivalence checking of two SQL queries is an intractable problem encountered in diverse contexts ranging from grading student submissions in a DBMS course to debugging query rewriting rules in an optimizer, and many more. While a lot of progress has been made in recent years in developing practical solutions for this problem, the existing methods can handle only a small subset of SQL, even for bounded equivalence checking. They cannot support sophisticated SQL expressions one encounters in practice. At the same time, large language models (LLMs) -- such as GPT-4 -- have emerged as power generators of SQL from natural language specifications. This paper explores whether LLMs can also demonstrate the ability to reason with SQL queries and help advance SQL equivalence checking. Towards this, we conducted a detailed evaluation of several LLMs over collections with SQL pairs of varying levels of complexity. We explored the efficacy of different prompting techniques, the utility of synthetic examples & explanations, as well as logical plans generated by query parsers. Our main finding is that with well-designed prompting using an unoptimized SQL Logical Plan, LLMs can perform equivalence checking beyond the capabilities of current techniques, achieving nearly 100% accuracy for equivalent pairs and up to 70% for non-equivalent pairs of SQL queries. While LLMs lack the ability to generate formal proofs, their synthetic examples and human-readable explanations offer valuable insights to students (& instructors) in a classroom setting and to database administrators (DBAs) managing large database installations. Additionally, we also show that with careful fine-tuning, we can close the performance gap between smaller (and efficient) models and larger models such as GPT, thus paving the way for potential LLM-integration in standalone data processing systems.

Exploring the Use of LLMs for SQL Equivalence Checking

LLM-SQL-Solver: Can LLMs Determine SQL Equivalence?

Proving Query Equivalence Using Linear Integer Arithmetic

Evaluating SQL Understanding in Large Language Models

Can LLMs substitute SQL? Comparing Resource Utilization of Querying LLMs versus Traditional Relational Databases

From Natural Language to SQL: Review of LLM-based Text-to-SQL Systems

Understanding Help-Seeking Behavior of Students Using LLMs vs. Web Search for Writing SQL Queries

Querying Large Language Models with SQL

Enhancing LLM Fine-tuning for Text-to-SQLs by SQL Quality Measurement

Lucy: Think and Reason to Solve Text-to-SQL

Evaluating LLMs for Text-to-SQL Generation With Complex SQL Workload

Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types

Analyzing the Effectiveness of Large Language Models on Text-to-SQL Synthesis

The Unreasonable Effectiveness of LLMs for Query Optimization

Optimizing LLM Queries in Relational Workloads

SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL

Logical Consistency of Large Language Models in Fact-checking

A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters

SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)

Can LLMs Compute with Reasons?

Using LLMs in Software Requirements Specifications: An Empirical Evaluation