Hybrid Querying Over Relational Databases and Large Language Models

Fuheng Zhao,Divyakant Agrawal,Amr El Abbadi

2024-08-02

Abstract:Database queries traditionally operate under the closed-world assumption, providing no answers to questions that require information beyond the data stored in the database. Hybrid querying using SQL offers an alternative by integrating relational databases with large language models (LLMs) to answer beyond-database questions. In this paper, we present the first cross-domain benchmark, SWAN, containing 120 beyond-database questions over four real-world databases. To leverage state-of-the-art language models in addressing these complex questions in SWAN, we present, HQDL, a preliminary solution for hybrid querying, and also discuss potential future directions. Our evaluation demonstrates that HQDL using GPT-4 Turbo with few-shot prompts, achieves 40.0\% in execution accuracy and 48.2\% in data factuality. These results highlights both the potential and challenges for hybrid querying. We believe that our work will inspire further research in creating more efficient and accurate data systems that seamlessly integrate relational databases and large language models to address beyond-database questions.

Databases,Computation and Language

What problem does this paper attempt to address?

The paper primarily focuses on addressing how to leverage large language models (LLMs) in conjunction with relational databases to answer questions that go beyond the information stored within the database itself (referred to as "beyond-database" questions). Traditionally, database queries are conducted under the closed-world assumption, meaning they provide answers based solely on the data stored in the database. However, in many cases, users may need to obtain answers based on information both inside and outside the database.

To achieve this goal, the authors propose the following points:

1. **SWAN Benchmark**: This is the first cross-domain benchmark set, containing 120 "beyond-database" questions for four real-world databases. These databases cover different domains, such as European football, Formula 1 racing, etc.

2. **HQDL Solution**: This is an initial approach to solving the complex questions mentioned above by integrating large language models with relational databases. HQDL includes the processes of data generation, extraction, and ultimately executing hybrid queries to answer user questions.

3. **Evaluation Results**: The authors evaluated using state-of-the-art language models like GPT-4 Turbo and reported results in terms of execution accuracy and data factuality. The results indicate that despite challenges, this approach shows potential.

Specifically, HQDL uses zero-shot and few-shot prompts to guide large language models in generating the required data. Experimental results show that when more examples are provided, the scores for execution accuracy and data factuality improve. Additionally, the paper discusses some limitations of HQDL and suggests directions for future improvements.

In summary, this paper explores how to combine large language models with relational databases to address questions that require information from both inside and outside the database. It proposes an initial solution and a benchmark set to promote further research in this field.