Abstract:As humans, we can naturally break down a task into individual steps in our daily lives and we are able to provide feedback or dynamically adjust the plan when encountering obstacles. Similarly, our aim is to facilitate agents in comprehending and carrying out natural language instructions in a more efficient and cost-effective manner. For example, in Vision-Language Navigation (VLN) tasks, the agent needs to understand instructions such as "go to the table by the fridge". This understanding allows the agent to navigate to the table and infer that the destination is likely to be in the kitchen. The traditional VLN approach mainly involves training models using a large number of labeled datasets for task planning in unseen environments. However, manual labeling incurs a high cost for this approach. Considering that large language models (LLMs) already possess extensive commonsense knowledge during pre-training, some researchers have started using LLMs as decision modules in embodied tasks, although this approach shows the LLMs' reasoning ability to plan a logical sequence of subtasks based on global information. However, executing subtasks often encounters issues, such as obstacles that hinder progress and alterations in the state of the target object. Even one mistake can cause the subsequent tasks to fail, which makes it challenging to complete the instructions through a single plan. Therefore, we propose a new approach-C (Correction) and P (Planning) with M (Memory) I (Integration)-that centered on an LLM for embodied tasks. In more detail, the auxiliary modules of the CPMI facilitate dynamic planning by the LLM-centric planner. These modules provide the agent with memory and generalized experience mechanisms to fully utilize the LLM capabilities, allowing it to improve its performance during execution. Finally, the experimental results on public datasets demonstrate that we achieve the best performance in the few-shot scenario, improving the efficiency of the successive task while increasing the success rate.

"My agent understands me better": Integrating Dynamic Human-like Memory Recall and Consolidation in LLM-Based Agents

Empowering Working Memory for Large Language Model Agents

Integrating Temporal Representations for Dynamic Memory Retrieval and Management in Large Language Models

RecallM: An Adaptable Memory Mechanism with Temporal Understanding for Large Language Models

Memory Sharing for Large Language Model based Agents

On the Structural Memory of LLM Agents

MemoryBank: Enhancing Large Language Models with Long-Term Memory

From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Models

Hello Again! LLM-powered Personalized Agent for Long-term Dialogue

HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model

Memory Matters: The Need to Improve Long-Term Memory in LLM-Agents

Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory

LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

Self-evolving Agents with reflective and memory-augmented abilities

Towards Lifelong Dialogue Agents via Relation-aware Memory Construction and Timeline-augmented Response Generation

Cooperative Scheduling and Hierarchical Memory Model for Multi-Agent Systems

Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents

Large Language Models Are Semi-Parametric Reinforcement Learning Agents

LDM$^2$: A Large Decision Model Imitating Human Cognition with Dynamic Memory Enhancement

Leave It to Large Language Models! Correction and Planning with Memory Integration