CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planning

Duo Wu,Jinghe Wang,Yuan Meng,Yanning Zhang,Le Sun,Zhi Wang

2024-11-25

Abstract:Utilizing large language models (LLMs) for tool planning has emerged as a promising avenue for developing general AI systems, where LLMs automatically schedule external tools (e.g. vision models) to tackle complex tasks based on task descriptions. To push this paradigm toward practical applications, it is crucial for LLMs to consider tool execution costs (e.g. execution time) for tool planning. Unfortunately, prior studies overlook the tool execution costs, leading to the generation of expensive plans of which the costs outweigh task performance. To fill this gap, we propose the Cost-Aware Tool Planning with LLMs (CATP-LLM) framework, which for the first time provides a coherent design to empower LLMs for cost-aware tool planning. Specifically, CATP-LLM incorporates a tool planning language to enhance the LLM to generate non-sequential plans of multiple branches for efficient concurrent tool execution and cost reduction. Moreover, it further designs a cost-aware offline reinforcement learning algorithm to fine-tune the LLM to optimize the performance-cost trade-off in tool planning. In lack of public cost-related datasets, we further present OpenCATP, the first platform for cost-aware planning evaluation. Experiments on OpenCATP show that CATP-LLM outperforms GPT-4 even when using Llama2-7B as its backbone, with the average improvement of 28.2%-30.2% higher plan performance and 24.7%-45.8% lower costs even on the challenging planning tasks. The codes of CATP-LLM and OpenCATP will be publicly available.

Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to consider the tool execution cost when using large - language models (LLMs) for tool planning, in order to achieve a good balance between performance and cost. Specifically, most of the existing research only supports sequential planning and ignores the cost of tool execution, which leads to the generated plans being potentially costly, and their execution costs may even exceed the value of task performance. To solve this problem, the paper proposes a new framework named CATP - LLM, which for the first time provides a coherent design to enhance the cost - aware tool - planning ability of LLMs. CATP - LLM, by introducing the Tool Planning Language (TPL) and the Cost - Aware Offline Reinforcement Learning algorithm (CAORL), not only supports the generation of non - sequential plans, but also optimizes the trade - off between performance and cost in tool planning. In addition, the paper also develops a new evaluation platform, OpenCATP, for comprehensively evaluating the performance of LLMs in cost - aware tool planning.