LLM+Reasoning+Planning for supporting incomplete user queries in presence of APIs

Sudhir Agarwal,Anu Sreepathy,David H. Alonso,Prarit Lamba

2024-10-11

Abstract:Recent availability of Large Language Models (LLMs) has led to the development of numerous LLM-based approaches aimed at providing natural language interfaces for various end-user tasks. These end-user tasks in turn can typically be accomplished by orchestrating a given set of APIs. In practice, natural language task requests (user queries) are often incomplete, i.e., they may not contain all the information required by the APIs. While LLMs excel at natural language processing (NLP) tasks, they frequently hallucinate on missing information or struggle with orchestrating the APIs. The key idea behind our proposed approach is to leverage logical reasoning and classical AI planning along with an LLM for accurately answering user queries including identification and gathering of any missing information in these queries. Our approach uses an LLM and ASP (Answer Set Programming) solver to translate a user query to a representation in Planning Domain Definition Language (PDDL) via an intermediate representation in ASP. We introduce a special API "get_info_api" for gathering missing information. We model all the APIs as PDDL actions in a way that supports dataflow between the APIs. Our approach then uses a classical AI planner to generate an orchestration of API calls (including calls to get_info_api) to answer the user query. Our evaluation results show that our approach significantly outperforms a pure LLM based approach by achieving over 95\% success rate in most cases on a dataset containing complete and incomplete single goal and multi-goal queries where the multi-goal queries may or may not require dataflow among the APIs.

Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to support incomplete user queries in the presence of APIs. Specifically, although large - language models (LLMs) perform well in natural - language - processing tasks, when dealing with incomplete user queries, they are often unable to accurately identify and collect missing information, or it is difficult to effectively coordinate API calls. The paper proposes a method that combines logical reasoning and classical AI planning to overcome these limitations and improve the ability to handle incomplete user queries. The main contributions of the paper include: 1. **Combining LLM with Logical Reasoning**: By using LLM to generate an intermediate representation of the user query and leveraging Answer Set Programming (ASP) for logical reasoning, to determine additional information, detect inconsistencies in the user query, and bridge the syntactic and semantic differences between the user query and the goal - task PDDL. 2. **Introducing a Special API**: A special `get_info_api` API is defined to collect missing information from the user or external systems at runtime, thus supporting the handling of incomplete queries. 3. **Using a Classical AI Planner**: Transforming the user query into a task - PDDL representation and using a classical AI planner to generate an orchestration plan for API calls to accurately answer the user query. 4. **Evaluation and Experiment**: By generating a dataset containing single - goal and multi - goal queries, the proposed framework was evaluated, and the results show that this method is significantly better than the pure LLM method in most cases, with a success rate of over 95%. Through these methods, the paper aims to provide a more efficient, accurate, and scalable solution to deal with the common problem of incomplete user queries in practical applications.

LLM+Reasoning+Planning for supporting incomplete user queries in presence of APIs

LLM+P: Empowering Large Language Models with Optimal Planning Proficiency

Planning in the Dark: LLM-Symbolic Planning Pipeline without Experts

Query-Efficient Planning with Language Models

Can LLMs Fix Issues with Reasoning Models? Towards More Likely Models for AI Planning

LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning

CLMASP: Coupling Large Language Models with Answer Set Programming for Robotic Task Planning

LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks

LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planning

Furthest Reasoning with Plan Assessment: Stable Reasoning Path with Retrieval-Augmented Large Language Models

Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models

On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)

Reliable Natural Language Understanding with Large Language Models and Answer Set Programming

Human-Centered Planning

ISR-LLM: Iterative Self-Refined Large Language Model for Long-Horizon Sequential Task Planning

Cooperative Strategic Planning Enhances Reasoning Capabilities in Large Language Models

Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks

Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning

LLM-based Frameworks for API Argument Filling in Task-Oriented Conversational Systems

Translating Natural Language to Planning Goals with Large-Language Models

NL2Plan: Robust LLM-Driven Planning from Minimal Text Descriptions