TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs

Yaobo Liang,Chenfei Wu,Ting Song,Wenshan Wu,Yan Xia,Yu Liu,Yang Ou,Shuai Lu,Lei Ji,Shaoguang Mao,Yun Wang,Linjun Shou,Ming Gong,Nan Duan
2023-03-29
Abstract:Artificial Intelligence (AI) has made incredible progress recently. On the one hand, advanced foundation models like ChatGPT can offer powerful conversation, in-context learning and code generation abilities on a broad range of open-domain tasks. They can also generate high-level solution outlines for domain-specific tasks based on the common sense knowledge they have acquired. However, they still face difficulties with some specialized tasks because they lack enough domain-specific data during pre-training or they often have errors in their neural network computations on those tasks that need accurate executions. On the other hand, there are also many existing models and systems (symbolic-based or neural-based) that can do some domain-specific tasks very well. However, due to the different implementation or working mechanisms, they are not easily accessible or compatible with foundation models. Therefore, there is a clear and pressing need for a mechanism that can leverage foundation models to propose task solution outlines and then automatically match some of the sub-tasks in the outlines to the off-the-shelf models and systems with special functionalities to complete them. Inspired by this, we introduce <a class="link-external link-http" href="http://TaskMatrix.AI" rel="external noopener nofollow">this http URL</a> as a new AI ecosystem that connects foundation models with millions of APIs for task completion. Unlike most previous work that aimed to improve a single AI model, <a class="link-external link-http" href="http://TaskMatrix.AI" rel="external noopener nofollow">this http URL</a> focuses more on using existing foundation models (as a brain-like central system) and APIs of other AI models and systems (as sub-task solvers) to achieve diversified tasks in both digital and physical domains. As a position paper, we will present our vision of how to build such an ecosystem, explain each key component, and use study cases to illustrate both the feasibility of this vision and the main challenges we need to address next.
Artificial Intelligence,Computation and Language
What problem does this paper attempt to address?
The problem that this paper attempts to solve is that although existing basic models (such as ChatGPT) perform well in a wide range of tasks, they still have difficulties in performing some tasks in specific fields. This is because these models lack sufficient domain - specific data or have neural network calculation errors in tasks that require precise execution. On the other hand, there are many existing models and systems (based on symbolic rules or neural networks) that perform well in certain specific tasks, but due to different implementations or working mechanisms, they are incompatible with basic models or difficult to access. To overcome these problems, the paper proposes a new AI ecosystem named TaskMatrix.AI. This system aims to complete diverse tasks by connecting basic models with millions of APIs, not limited to the digital field, but also including tasks in the physical world. The core advantage of TaskMatrix.AI lies in its ability to use basic models to understand different types of inputs and generate code for calling APIs to complete specific tasks; at the same time, it also has strong lifelong learning ability and can expand its skills by adding new APIs to handle new tasks. In addition, TaskMatrix.AI improves the interpretability of responses because it not only provides the task - solving logic (i.e., action code) but also the understandability of API results. In this way, TaskMatrix.AI aims to build an ecosystem that can make full use of basic models and other models and systems that are good at specific tasks, thereby achieving efficient completion of a wider range of digital and physical tasks.