Abstract:Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.

What problem does this paper attempt to address?

The problem this paper attempts to address is how to make large language models (LLMs) more effective in reasoning and decision-making in the real world. Specifically, the authors propose an embodied multimodal language model called PaLM-E, which aims to directly integrate continuous sensor modality inputs (such as images, state estimates, or other sensor data) from embodied experience agents into the language model. This enables more natural language learning and the ability to solve a range of embodied reasoning tasks. ### Main Issues 1. **Reasoning and Decision-Making in the Real World**: - Although current large language models exhibit strong reasoning capabilities across various tasks, their application in the real world is limited by the "grounding" problem. That is, these models struggle to connect textual representations with real-world visual and physical sensor modalities, thereby addressing a broader range of practical problems based on computer vision and robotics. 2. **Design and Training of Embodied Language Models**: - How to design and train a large language model capable of handling multimodal inputs (including text, images, and other sensor data) so that it performs well in tasks such as robotic planning, visual question answering, and image description. ### Solution - **Embodied Multimodal Language Model (PaLM-E)**: - PaLM-E processes multimodal inputs by embedding continuous sensor inputs (such as images and state estimates) into the embedding space of the language model. These inputs are encoded into vectors of the same dimension as language tokens and processed together with text tokens through self-attention layers. - The model can generate natural language instructions for various embodied tasks, which can be interpreted and executed by low-level policies or actuators, enabling high-level decision-making and control. - **Multitask Training**: - The authors improve the model's performance through multitask training, demonstrating that this training method can significantly enhance data efficiency and even achieve one-shot or zero-shot generalization to new objects or task combinations. - **Large-Scale Model**: - The largest version of PaLM-E (PaLM-E-562B) integrates the 54 billion parameter PaLM and the 2 billion parameter Vision Transformer (ViT), making it the largest reported vision-language model to date. This model achieved state-of-the-art performance on the OK-VQA benchmark without task-specific fine-tuning. ### Main Contributions 1. **Proposing and Validating the Effectiveness of an Embodied Multimodal Language Model**: - Demonstrated that by incorporating embodied experience data into the training of a multimodal large language model, a general model capable of making decisions across multiple embodied experiences can be trained. 2. **Showing the Limitations of Current State-of-the-Art Vision-Language Models in Zero-Shot Embodied Reasoning**: - However, with appropriate training, a model that excels in both vision-language tasks and efficient embodied reasoning can be constructed. 3. **Introducing New Architectural Ideas**: - Including neural scene representations and entity-tagged multimodal tokens, these innovations help better handle multimodal inputs. 4. **Demonstrating PaLM-E's Quantitative Capabilities in Vision and Language Tasks**: - Besides being an embodied reasoning model, PaLM-E also performs well in vision and language tasks. 5. **Proving that Increasing the Scale of the Language Model Can Reduce Catastrophic Forgetting**: - During multimodal fine-tuning, larger language models can better retain learned knowledge. In summary, this paper addresses the problem of applying large language models to embodied reasoning and decision-making in the real world by proposing the PaLM-E model, providing an important reference and direction for future multimodal AI research.

PaLM-E: An Embodied Multimodal Language Model

MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World

AudioPaLM: A Large Language Model That Can Speak and Listen

Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond

Language Models Meet World Models: Embodied Experiences Enhance Language Models

eP-ALM: Efficient Perceptual Augmentation of Language Models

EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model

PALO: A Polyglot Large Multimodal Model for 5B People

PAL: Program-aided Language Models

PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain

LaMI: Large Language Models for Multi-Modal Human-Robot Interaction

Visually-Augmented Language Modeling

InfMLLM: A Unified Framework for Visual-Language Tasks.

PaLM: Scaling Language Modeling with Pathways

Language Is Not All You Need: Aligning Perception with Language Models

Grounding Large Language Models In Embodied Environment With Imperfect World Models

Empowering MultiModal Models' In-Context Learning Ability through Large Language Models.