Diffusion Models as Optimizers for Efficient Planning in Offline RL

Renming Huang,Yunqiang Pei,Guoqing Wang,Yangming Zhang,Yang Yang,Peng Wang,Hengtao Shen

2024-07-23

Abstract:Diffusion models have shown strong competitiveness in offline reinforcement learning tasks by formulating decision-making as sequential generation. However, the practicality of these methods is limited due to the lengthy inference processes they require. In this paper, we address this problem by decomposing the sampling process of diffusion models into two decoupled subprocesses: 1) generating a feasible trajectory, which is a time-consuming process, and 2) optimizing the trajectory. With this decomposition approach, we are able to partially separate efficiency and quality factors, enabling us to simultaneously gain efficiency advantages and ensure quality assurance. We propose the Trajectory Diffuser, which utilizes a faster autoregressive model to handle the generation of feasible trajectories while retaining the trajectory optimization process of diffusion models. This allows us to achieve more efficient planning without sacrificing capability. To evaluate the effectiveness and efficiency of the Trajectory Diffuser, we conduct experiments on the D4RL benchmarks. The results demonstrate that our method achieves $\it 3$-$\it 10 \times$ faster inference speed compared to previous sequence modeling methods, while also outperforming them in terms of overall performance. <a class="link-external link-https" href="https://github.com/RenMing-Huang/TrajectoryDiffuser" rel="external noopener nofollow">this https URL</a> Keywords: Reinforcement Learning and Efficient Planning and Diffusion Model

Machine Learning,Artificial Intelligence,Computer Vision and Pattern Recognition,Robotics

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper aims to address the issue of the prolonged inference time of diffusion models in offline reinforcement learning (Offline RL). Specifically, although diffusion models demonstrate strong competitiveness in offline reinforcement learning tasks, their lengthy denoising process limits their practical application in rapidly changing environments, such as autonomous driving scenarios.

#### Main Contributions
1. **Decomposing the Inference Process**: The inference process of the diffusion model is decomposed into two steps: generating feasible trajectories and optimizing trajectories. This decomposition allows efficiency and quality factors to be considered separately, thereby improving efficiency while ensuring the quality of the trajectories.
2. **Introducing the Trajectory Diffuser**: A new offline reinforcement learning algorithm is proposed, which not only alleviates the computational challenges of using diffusion models for trajectory planning but also significantly enhances performance.
3. **Experimental Evaluation**: The efficiency and performance of the Trajectory Diffuser are evaluated on D4RL benchmark tasks, showing a 3-10 times improvement in efficiency compared to previous diffusion model-based methods, and also surpassing them in performance.

Through these improvements, the paper aims to achieve more efficient planning without sacrificing performance.