ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrieval

Yuanhang Zheng,Peng Li,Wei Liu,Yang Liu,Jian Luan,Bin Wang

DOI: https://doi.org/10.48550/arXiv.2403.06551

2024-03-11

Abstract:Tool learning aims to extend the capabilities of large language models (LLMs) with external tools. A major challenge in tool learning is how to support a large number of tools, including unseen tools. To address this challenge, previous studies have proposed retrieving suitable tools for the LLM based on the user query. However, previously proposed methods do not consider the differences between seen and unseen tools, nor do they take the hierarchy of the tool library into account, which may lead to suboptimal performance for tool retrieval. Therefore, to address the aforementioned issues, we propose ToolRerank, an adaptive and hierarchy-aware reranking method for tool retrieval to further refine the retrieval results. Specifically, our proposed ToolRerank includes Adaptive Truncation, which truncates the retrieval results related to seen and unseen tools at different positions, and Hierarchy-Aware Reranking, which makes retrieval results more concentrated for single-tool queries and more diverse for multi-tool queries. Experimental results show that ToolRerank can improve the quality of the retrieval results, leading to better execution results generated by the LLM.

Information Retrieval

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve the main challenges encountered by large - language models (LLMs) in tool learning, that is, how to support a large number of tools, including unseen tools. Specifically, the existing methods have the following problems when retrieving tools suitable for LLMs:

1. **Distinguishing between seen and unseen tools**: The existing methods do not take into account the differences between seen and unseen tools, which may lead to poor retrieval performance.
2. **Hierarchical structure of the tool library**: The existing methods do not consider the hierarchical structure of the tool library, which may result in the retrieval results of single - tool queries being too scattered and the retrieval results of multi - tool queries not being diverse enough.

To address these problems, the authors propose **ToolRerank**, an adaptive and hierarchy - aware re - ranking method for further optimizing tool retrieval results. ToolRerank includes two key components:

1. **Adaptive Truncation**: Truncate the retrieval results at different positions according to whether the query is related to seen or unseen tools.
2. **Hierarchy - Aware Reranking**: Further optimize the retrieval results according to the query type (single - tool query or multi - tool query), making the results of single - tool queries more concentrated and the results of multi - tool queries more diverse.

Through these methods, ToolRerank can improve the quality of retrieval results, thereby generating better execution results. Experimental results show that ToolRerank outperforms the baseline methods on multiple test datasets.