Abstract:Large Language Models (LLMs) have demonstrated remarkable abilities in various language tasks, making them promising candidates for decision-making in robotics. Inspired by Hierarchical Reinforcement Learning (HRL), we propose Retrieval-Augmented in-context reinforcement Learning (RAHL), a novel framework that decomposes complex tasks into sub-tasks using an LLM-based high-level policy, in which a complex task is decomposed into sub-tasks by a high-level policy on-the-fly. The sub-tasks, defined by goals, are assigned to the low-level policy to complete. To improve the agent's performance in multi-episode execution, we propose Hindsight Modular Reflection (HMR), where, instead of reflecting on the full trajectory, we let the agent reflect on shorter sub-trajectories to improve reflection efficiency. We evaluated the decision-making ability of the proposed RAHL in three benchmark environments--ALFWorld, Webshop, and HotpotQA. The results show that RAHL can achieve an improvement in performance in 9%, 42%, and 10% in 5 episodes of execution in strong baselines. Furthermore, we also implemented RAHL on the Boston Dynamics SPOT robot. The experiment shows that the robot can scan the environment, find entrances, and navigate to new rooms controlled by the LLM policy.

What problem does this paper attempt to address?

The paper attempts to address the problem of how to efficiently utilize large language models (LLMs) for task decomposition and decision-making in multi-stage task planning, and improve the model's performance in multi-round execution through a multi-level reflection mechanism. Specifically: 1. **Task Decomposition and Decision-Making**: The paper proposes a new framework—Retrieval-Augmented Hierarchical in-context Reinforcement Learning (RAHL). This framework uses LLMs to decompose complex tasks into subtasks and accomplish these subtasks through the collaboration of high-level and low-level policies. The high-level policy is responsible for generating sub-goals, while the low-level policy is responsible for achieving these sub-goals. 2. **Performance Improvement in Multi-Round Execution**: To enhance the agent's performance in multi-round execution, the paper introduces the Hindsight Modular Reflection (HMR) mechanism. HMR decomposes the reflection process into two parts: low-level reflection focuses on the implementation process of each sub-goal, while high-level reflection focuses on the entire sequence of sub-goals. This modular reflection method helps the agent identify errors and optimize strategies. 3. **Experimental Validation**: The paper conducts experimental validation in three benchmark environments, including ALFWorld, Webshop, and HotpotQA. The experimental results show that RAHL significantly outperforms existing baseline methods in these environments, especially in multi-round execution. 4. **Hardware Experiments**: In addition to experiments in virtual environments, the paper also demonstrates the application of RAHL on the Boston Dynamics SPOT robot, proving the effectiveness and feasibility of this method in real robotic tasks. In summary, the paper aims to improve the decision-making ability and performance in multi-round execution of agents in complex task planning by combining LLMs and a multi-level reflection mechanism.

Retrieval-Augmented Hierarchical in-Context Reinforcement Learning and Hindsight Modular Reflections for Task Planning with LLMs

FLTRNN: Faithful Long-Horizon Task Planning for Robotics with Large Language Models

ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL

Goal-Conditioned Hierarchical Reinforcement Learning with High-Level Model Approximation.

LLM Augmented Hierarchical Agents

LGR2: Language Guided Reward Relabeling for Accelerating Hierarchical Reinforcement Learning

Guiding Multi-agent Multi-task Reinforcement Learning by a Hierarchical Framework with Logical Reward Shaping

Planning-Augmented Hierarchical Reinforcement Learning

RePLan: Robotic Replanning with Perception and Language Models

HCS-R-HER: Hierarchical Reinforcement Learning Based on Cross Subtasks Rainbow Hindsight Experience Replay

Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks

Hierarchies of Planning and Reinforcement Learning for Robot Navigation

Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Representation

SRLM: Human-in-Loop Interactive Social Robot Navigation with Large Language Model and Deep Reinforcement Learning

RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents

Exploring the limits of Hierarchical World Models in Reinforcement Learning

RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models

Data-Efficient Hierarchical Reinforcement Learning for Robotic Assembly Control Applications

Hierarchical reinforcement learning for handling sparse rewards in multi-goal navigation

Hierarchical reinforcement learning with natural language subgoals