Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

Yijun Yang,Tianyi Zhou,Kanxue Li,Dapeng Tao,Lusong Li,Li Shen,Xiaodong He,Jing Jiang,Yuhui Shi

2024-03-29

Abstract:While large language models (LLMs) excel in a simulated world of texts, they struggle to interact with the more realistic world without perceptions of other modalities such as visual or audio signals. Although vision-language models (VLMs) integrate LLM modules (1) aligned with static image features, and (2) may possess prior knowledge of world dynamics (as demonstrated in the text world), they have not been trained in an embodied visual world and thus cannot align with its dynamics. On the other hand, training an embodied agent in a noisy visual world without expert guidance is often challenging and inefficient. In this paper, we train a VLM agent living in a visual world using an LLM agent excelling in a parallel text world. Specifically, we distill LLM's reflection outcomes (improved actions by analyzing mistakes) in a text world's tasks to finetune the VLM on the same tasks of the visual world, resulting in an Embodied Multi-Modal Agent (EMMA) quickly adapting to the visual world dynamics. Such cross-modality imitation learning between the two parallel worlds is achieved by a novel DAgger-DPO algorithm, enabling EMMA to generalize to a broad scope of new tasks without any further guidance from the LLM expert. Extensive evaluations on the ALFWorld benchmark's diverse tasks highlight EMMA's superior performance to SOTA VLM-based agents, e.g., 20%-70% improvement in the success rate.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper discusses how to train an Embodied Multi-Modal Agent (EMMA) to match the performance of a Large Language Model (LLM) in the textual world within a visual world. Currently, while LLMs perform well in the textual world, they struggle with interacting in a more realistic world that includes visual or audio signals. Although Visual Language Models (VLMs) attempt to align LLMs with visual patterns, their pre-training only focuses on aligning static image-text pairs and does not fully consider the dynamics of the visual world.

The paper proposes a novel approach, which fine-tunes a VLM in the visual world by leveraging feedback from LLM experts in the textual world through cross-modal interactive imitation learning. Specifically, the visual observations of the VLM are transformed into equivalent textual descriptions, and then actions that the VLM can imitate are generated by the LLM experts. This approach, based on the DAgger-DPO algorithm, overcomes the accumulated error and distribution shift problems of behavior cloning.

Experiments conducted on the ALFWorld benchmark test show that EMMA significantly outperforms other VLM-based agents in terms of success rate, improving by 20% to 70%, and is the only VLM agent capable of generalizing to open-vocabulary and unconstrained testing tasks. Furthermore, the research also found that EMMA is more robust in adapting to noisy visual inputs compared to agents solely based on language models, demonstrating its resilience in a multi-modal environment.

The main contribution of the paper is the proposal of a method to train a more flexible and generalizable multi-modal agent through LLM feedback, which is of significant importance for achieving Artificial General Intelligence (AGI).