Abstract:Large language models (LLMs) have garnered significant attention due to their impressive natural language processing (NLP) capabilities. Recently, many studies have focused on the tool utilization ability of LLMs. They primarily investigated how LLMs effectively collaborate with given specific tools. However, in scenarios where LLMs serve as intelligent agents, as seen in applications like AutoGPT and MetaGPT, LLMs are expected to engage in intricate decision-making processes that involve deciding whether to employ a tool and selecting the most suitable tool(s) from a collection of available tools to fulfill user requests. Therefore, in this paper, we introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools. Specifically, we create a dataset called ToolE within the benchmark. This dataset contains various types of user queries in the form of prompts that trigger LLMs to use tools, including both single-tool and multi-tool scenarios. Subsequently, we set the tasks for both tool usage awareness and tool selection. We define four subtasks from different perspectives in tool selection, including tool selection with similar choices, tool selection in specific scenarios, tool selection with possible reliability issues, and multi-tool selection. We conduct experiments involving eight popular LLMs and find that the majority of them still struggle to effectively select tools, highlighting the existing gaps between LLMs and genuine intelligent agents. However, through the error analysis, we found there is still significant room for improvement. Finally, we conclude with insights for tool developers -- we strongly recommend that tool developers choose an appropriate rewrite model for generating new descriptions based on the downstream LLM the tool will apply to. Our code is in <a class="link-external link-https" href="https://github.com/HowieHwong/MetaTool" rel="external noopener nofollow">this https URL</a>.

Toolken+: Improving LLM Tool Usage with Reranking and a Reject Option

ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings

ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrieval

Meta-Reasoning Improves Tool Use in Large Language Models

ToolTalk: Evaluating Tool-Usage in a Conversational Setting

MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use

Efficient and Scalable Estimation of Tool Representations in Vector Space

A Comprehensive Evaluation of Tool-Assisted Generation Strategies

StepTool: A Step-grained Reinforcement Learning Framework for Tool Learning in LLMs

PTR: Precision-Driven Tool Recommendation for Large Language Models

WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models

RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning

ProTIP: Progressive Tool Retrieval Improves Planning

Planning and Editing What You Retrieve for Enhanced Tool Learning

Tool-Planner: Task Planning with Clusters across Multiple Tools

LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error

What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks

GeckOpt: LLM System Efficiency via Intent-Based Tool Selection

Toolshed: Scale Tool-Equipped Agents with Advanced RAG-Tool Fusion and Tool Knowledge Bases

ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios