Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making

Manling Li,Shiyu Zhao,Qineng Wang,Kangrui Wang,Yu Zhou,Sanjana Srivastava,Cem Gokmen,Tony Lee,Li Erran Li,Ruohan Zhang,Weiyu Liu,Percy Liang,Li Fei-Fei,Jiayuan Mao,Jiajun Wu
2024-10-10
Abstract:We aim to evaluate Large Language Models (LLMs) for embodied decision making. While a significant body of work has been leveraging LLMs for decision making in embodied environments, we still lack a systematic understanding of their performance because they are usually applied in different domains, for different purposes, and built based on different inputs and outputs. Furthermore, existing evaluations tend to rely solely on a final success rate, making it difficult to pinpoint what ability is missing in LLMs and where the problem lies, which in turn blocks embodied agents from leveraging LLMs effectively and selectively. To address these limitations, we propose a generalized interface (Embodied Agent Interface) that supports the formalization of various types of tasks and input-output specifications of LLM-based modules. Specifically, it allows us to unify 1) a broad set of embodied decision-making tasks involving both state and temporally extended goals, 2) four commonly-used LLM-based modules for decision making: goal interpretation, subgoal decomposition, action sequencing, and transition modeling, and 3) a collection of fine-grained metrics which break down evaluation into various types of errors, such as hallucination errors, affordance errors, various types of planning errors, etc. Overall, our benchmark offers a comprehensive assessment of LLMs' performance for different subtasks, pinpointing the strengths and weaknesses in LLM-powered embodied AI systems, and providing insights for effective and selective use of LLMs in embodied decision making.
Computation and Language,Artificial Intelligence,Machine Learning,Robotics
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to evaluate the performance of large - language models (LLMs) in embodied decision - making. Although many studies have utilized LLMs for decision - making in embodied environments, there is currently a lack of systematic understanding of the performance of these models, because they are usually applied in different fields, for different purposes, and constructed based on different inputs and outputs. In addition, existing evaluation methods often rely solely on the final success rate, which makes it difficult to determine which capabilities LLMs lack and where the problems lie, thereby hindering embodied agents from effectively and selectively using LLMs. To address these limitations, the authors propose a general interface (EMBODIED AGENT INTERFACE) that supports the formalization of various task types and LLM module input - output specifications. Specifically, this interface allows for the unification of: 1. A series of embodied decision - making tasks involving states and time - extended goals. 2. Four commonly - used LLM - based decision - making modules: goal interpretation, sub - goal decomposition, action sequencing, and transition modeling. 3. A series of fine - grained evaluation metrics that decompose the evaluation into various types of errors, such as hallucination errors, operability errors, various planning errors, etc. Overall, this benchmark provides a comprehensive evaluation of the performance of LLMs in different subtasks, points out the advantages and disadvantages of LLM - driven embodied AI systems, and provides insights for the effective and selective use of LLMs in embodied decision - making.