Adaptive Online Replanning with Diffusion Models

Siyuan Zhou,Yilun Du,Shun Zhang,Mengdi Xu,Yikang Shen,Wei Xiao,Dit-Yan Yeung,Chuang Gan

2023-10-15

Abstract:Diffusion models have risen as a promising approach to data-driven planning, and have demonstrated impressive robotic control, reinforcement learning, and video planning performance. Given an effective planner, an important question to consider is replanning -- when given plans should be regenerated due to both action execution error and external environment changes. Direct plan execution, without replanning, is problematic as errors from individual actions rapidly accumulate and environments are partially observable and stochastic. Simultaneously, replanning at each timestep incurs a substantial computational cost, and may prevent successful task execution, as different generated plans prevent consistent progress to any particular goal. In this paper, we explore how we may effectively replan with diffusion models. We propose a principled approach to determine when to replan, based on the diffusion model's estimated likelihood of existing generated plans. We further present an approach to replan existing trajectories to ensure that new plans follow the same goal state as the original trajectory, which may efficiently bootstrap off previously generated plans. We illustrate how a combination of our proposed additions significantly improves the performance of diffusion planners leading to 38\% gains over past diffusion planning approaches on Maze2D, and further enables the handling of stochastic and long-horizon robotic control tasks. Videos can be found on the anonymous website: \url{<a class="link-external link-https" href="https://vis-www.cs.umass.edu/replandiffuser/" rel="external noopener nofollow">this https URL</a>}.

Robotics

What problem does this paper attempt to address?

The paper primarily aims to address the replanning problem encountered when using diffusion models for data-driven planning. Specifically:

1. **When to Replan**: In dynamic environments or in the presence of execution errors, directly executing the plan can lead to cumulative errors, making the plan infeasible. Traditional fixed-interval replanning methods, while simple, are not flexible enough and may result in unnecessary computational overhead. This paper proposes a method to determine when to replan based on the likelihood estimated by the diffusion model. When the likelihood of the current plan execution drops to a certain level, it indicates the need for replanning.

2. **How to Replan**: Three different replanning strategies are proposed:
- **Replanning from Scratch**: Completely regenerate a new plan, but this method is computationally expensive.
- **Replanning Based on Past Context**: Adjust only part of the future states, retaining the executed part to save computational resources.
- **Replanning Based on Future Context**: Remove the executed states and repeat the last predicted state to fill the remaining part, then perform a small amount of noise addition and denoising operations on this basis.

Experiments have validated the effectiveness of the proposed method (RDM) across different benchmarks, including long-horizon planning tasks (such as Maze2D), performance in random environments, and scenarios like robot control tasks. Notably, in the Maze2D environment, RDM achieved significant improvements compared to other baseline algorithms, with up to a 63% performance gain on large-scale maps.