An Interactive Agent Foundation Model
Zane Durante,Bidipta Sarkar,Ran Gong,Rohan Taori,Yusuke Noda,Paul Tang,Ehsan Adeli,Shrinidhi Kowshika Lakshmikanth,Kevin Schulman,Arnold Milstein,Demetri Terzopoulos,Ade Famoti,Noboru Kuno,Ashley Llorens,Hoi Vo,Katsu Ikeuchi,Li Fei-Fei,Jianfeng Gao,Naoki Wake,Qiuyuan Huang
2024-06-17
Abstract:The development of artificial intelligence systems is transitioning from creating static, task-specific models to dynamic, agent-based systems capable of performing well in a wide range of applications. We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents across a wide range of domains, datasets, and tasks. Our training paradigm unifies diverse pre-training strategies, including visual masked auto-encoders, language modeling, and next-action prediction, enabling a versatile and adaptable AI framework. We demonstrate the performance of our framework across three separate domains -- Robotics, Gaming AI, and Healthcare. Our model demonstrates its ability to generate meaningful and contextually relevant outputs in each area. The strength of our approach lies in its generality, leveraging a variety of data sources such as robotics sequences, gameplay data, large-scale video datasets, and textual information for effective multimodal and multi-task learning. Our approach provides a promising avenue for developing generalist, action-taking, multimodal systems.
Artificial Intelligence,Machine Learning,Robotics