Towards Completeness-Oriented Tool Retrieval for Large Language Models

Changle Qu,Sunhao Dai,Xiaochi Wei,Hengyi Cai,Shuaiqiang Wang,Dawei Yin,Jun Xu,Ji-Rong Wen

DOI: https://doi.org/10.1145/3627673.3679847

2024-07-29

Abstract:Recently, integrating external tools with Large Language Models (LLMs) has gained significant attention as an effective strategy to mitigate the limitations inherent in their pre-training data. However, real-world systems often incorporate a wide array of tools, making it impractical to input all tools into LLMs due to length limitations and latency constraints. Therefore, to fully exploit the potential of tool-augmented LLMs, it is crucial to develop an effective tool retrieval system. Existing tool retrieval methods primarily focus on semantic matching between user queries and tool descriptions, frequently leading to the retrieval of redundant, similar tools. Consequently, these methods fail to provide a complete set of diverse tools necessary for addressing the multifaceted problems encountered by LLMs. In this paper, we propose a novel modelagnostic COllaborative Learning-based Tool Retrieval approach, COLT, which captures not only the semantic similarities between user queries and tool descriptions but also takes into account the collaborative information of tools. Specifically, we first fine-tune the PLM-based retrieval models to capture the semantic relationships between queries and tools in the semantic learning stage. Subsequently, we construct three bipartite graphs among queries, scenes, and tools and introduce a dual-view graph collaborative learning framework to capture the intricate collaborative relationships among tools during the collaborative learning stage. Extensive experiments on both the open benchmark and the newly introduced ToolLens dataset show that COLT achieves superior performance. Notably, the performance of BERT-mini (11M) with our proposed model framework outperforms BERT-large (340M), which has 30 times more parameters. Furthermore, we will release ToolLens publicly to facilitate future research on tool retrieval.

Computation and Language,Information Retrieval

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively retrieve a complete and diverse set of tools when integrating external tools in large - language models (LLMs) to deal with multi - aspect problems. Existing tool retrieval methods mainly focus on the semantic matching between queries and tool descriptions, which often leads to the retrieval of redundant or similar tools and fails to provide the complete tool set required to handle the complex problems encountered by LLMs. Therefore, this paper proposes a new tool retrieval method based on collaborative learning - COLT (COllaborative Learning - based Tool Retrieval), aiming to improve the integrity of tool retrieval.

Specifically, COLT achieves this goal through two main stages: semantic learning and collaborative learning. In the semantic learning stage, the pre - trained language model is first fine - tuned to capture the semantic relationships between queries and tools. In the collaborative learning stage, three bipartite graphs (query - scene, query - tool, scene - tool) are constructed, and the complex collaborative relationships between tools are captured through message passing and cross - view graph contrastive learning. In addition, in order to ensure the balanced retrieval of diverse tools from the complete ground - truth tool set, a list - style multi - label loss is designed as the main learning objective loss.

The paper also introduces a new evaluation metric COMP@K for evaluating the integrity - based tool retrieval performance and constructs a new dataset ToolLens, in which each query usually requires the use of multiple related but diverse tools to solve, reflecting the multifaceted nature of user requests in the real world. These contributions jointly promote the research progress in the field of tool retrieval, especially for application scenarios that require multi - tool collaboration.