Abstract:Artificial Intelligence (AI) has made incredible progress recently. On the one hand, advanced foundation models like ChatGPT can offer powerful conversation, in-context learning and code generation abilities on a broad range of open-domain tasks. They can also generate high-level solution outlines for domain-specific tasks based on the common sense knowledge they have acquired. However, they still face difficulties with some specialized tasks because they lack enough domain-specific data during pre-training or they often have errors in their neural network computations on those tasks that need accurate executions. On the other hand, there are also many existing models and systems (symbolic-based or neural-based) that can do some domain-specific tasks very well. However, due to the different implementation or working mechanisms, they are not easily accessible or compatible with foundation models. Therefore, there is a clear and pressing need for a mechanism that can leverage foundation models to propose task solution outlines and then automatically match some of the sub-tasks in the outlines to the off-the-shelf models and systems with special functionalities to complete them. Inspired by this, we introduce <a class="link-external link-http" href="http://TaskMatrix.AI" rel="external noopener nofollow">this http URL</a> as a new AI ecosystem that connects foundation models with millions of APIs for task completion. Unlike most previous work that aimed to improve a single AI model, <a class="link-external link-http" href="http://TaskMatrix.AI" rel="external noopener nofollow">this http URL</a> focuses more on using existing foundation models (as a brain-like central system) and APIs of other AI models and systems (as sub-task solvers) to achieve diversified tasks in both digital and physical domains. As a position paper, we will present our vision of how to build such an ecosystem, explain each key component, and use study cases to illustrate both the feasibility of this vision and the main challenges we need to address next.

Sora for Foundation Robots with Parallel Intelligence: Three World Models, Three Robotic Systems

Sora-Based Parallel Vision for Smart Sensing of Intelligent Vehicles: from Foundation Models to Foundation Intelligence

Foundation models in robotics: Applications, challenges, and the future

When Does Sora Show: The Beginning of TAO to Imaginative Intelligence and Scenarios Engineering

Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis

A Survey on Robotics with Foundation Models: toward Embodied AI

Robot Learning in the Era of Foundation Models: A Survey

Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models

TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs

Foundation Models for Education: Promises and Prospects

A Taxonomy of Foundation Model based Systems through the Lens of Software Architecture

An Interactive Agent Foundation Model

A Reference Architecture for Designing Foundation Model based Systems

Real-World Robot Applications of Foundation Models: A Review

Training and Serving System of Foundation Models: A Comprehensive Survey

Foundation Model based Open Vocabulary Task Planning and Executive System for General Purpose Service Robots

Position Paper: Agent AI Towards a Holistic Intelligence

Sora and V-JEPA Have Not Learned The Complete Real World Model -- A Philosophical Analysis of Video AIs Through the Theory of Productive Imagination

Special issue on real-world robot applications of the foundation models