GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction

Rui Yang,Lin Song,Yanwei Li,Sijie Zhao,Yixiao Ge,Xiu Li,Ying Shan

2023-05-30

Abstract:This paper aims to efficiently enable Large Language Models (LLMs) to use multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have shown great potential for tool usage through sophisticated prompt engineering. Nevertheless, these models typically rely on prohibitive computational costs and publicly inaccessible data. To address these challenges, we propose the GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and OPT, to use tools. It generates an instruction-following dataset by prompting an advanced teacher with various multi-modal contexts. By using the Low-Rank Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs to solve a range of visual problems, including visual comprehension and image generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to use tools, which is performed in both zero-shot and fine-tuning ways. Extensive experiments demonstrate the effectiveness of our method on various language models, which not only significantly improves the accuracy of invoking seen tools, but also enables the zero-shot capacity for unseen tools. The code and demo are available at <a class="link-external link-https" href="https://github.com/StevenGrove/GPT4Tools" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to efficiently enable open - source large language models (such as LLaMA and OPT) to use multimodal tools. Although existing high - level proprietary large language models (such as ChatGPT and GPT - 4) have demonstrated great potential for using tools through complex prompt engineering, these models usually rely on high computational costs and data that are not accessible to the public. To overcome these challenges, the paper proposes the self - guided GPT4Tools method, aiming to enable open - source large language models to use tools. Specifically, GPT4Tools generates an instruction - following dataset by prompting a high - level teacher model and uses the low - rank adaptation (LoRA) optimization technique to fine - tune the open - source large language models to solve a series of visual problems, including visual understanding and image generation. In addition, the paper also provides a benchmark to evaluate the ability of large language models to use tools, which is tested in zero - shot and fine - tuning ways. The experimental results show that this method not only significantly improves the accuracy of calling seen tools, but also enables the calling of unseen tools in the zero - shot case.