ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph

Xukun Liu,Zhiyuan Peng,Xiaoyuan Yi,Xing Xie,Lirong Xiang,Yuchen Liu,Dongkuan Xu

2024-02-29

Abstract:While achieving remarkable progress in a broad range of tasks, large language models (LLMs) remain significantly limited in properly using massive external tools. Existing in-context learning approaches simply format tools into a list of plain text descriptions and input them to LLMs, from which, LLMs generate a sequence of tool calls to solve problems step by step. Such a paradigm ignores the intrinsic dependency between tools and offloads all reasoning loads to LLMs, making them restricted to a limited number of specifically designed tools. It thus remains challenging for LLMs to operate on a library of massive tools, casting a great limitation when confronted with real-world scenarios. This paper proposes ToolNet, a plug-and-play framework that scales up the number of tools to thousands with a moderate increase in token consumption. ToolNet organizes tools into a directed graph. Each node represents a tool, and weighted edges denote tool transition. Starting from an initial tool node, an LLM navigates in the graph by iteratively choosing the next one from its successors until the task is resolved. Extensive experiments show that ToolNet can achieve impressive results in challenging multi-hop tool learning datasets and is resilient to tool failures.

Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The paper mainly discusses how to enable large language models (LLMs) to effectively utilize a large amount of external tools. Existing methods transform tools into a text list input for LLMs, but this approach ignores the inherent dependencies between tools and places all reasoning work on LLMs, limiting their functionality. As the number of tools increases, the performance of LLMs in handling tasks will decrease because it becomes difficult for them to select the correct tool from a large number of options.

To address this issue, the paper proposes ToolNet, a scalable framework that organizes tools into a directed graph, allowing LLMs to start from an initial tool node and iteratively select subsequent tools until the problem is solved. ToolNet enhances the capability of LLMs to handle multi-step tool learning tasks and demonstrates certain robustness towards tool failures.

The paper analyzes the problems existing in current tool learning methods, such as excessive token consumption, inability to adapt to new tools or feature updates, and errors caused by low-quality tools. ToolNet reduces token consumption by selecting tools only from the subsequent tools of the previously chosen one, and dynamically adjusts the weights of tool transformations by evaluating their effectiveness, thus avoiding the use of low-quality tools.

Experiments show that ToolNet performs well on challenging multi-step tool learning datasets, and it has the ability to recover from tool failures while reducing token usage. A comparison with existing methods demonstrates the advantages of ToolNet in terms of performance and efficiency.