AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model

Zibin Dong,Yifu Yuan,Jianye Hao,Fei Ni,Yao Mu,Yan Zheng,Yujing Hu,Tangjie Lv,Changjie Fan,Zhipeng Hu

2024-02-04

Abstract:Aligning agent behaviors with diverse human preferences remains a challenging problem in reinforcement learning (RL), owing to the inherent abstractness and mutability of human preferences. To address these issues, we propose AlignDiff, a novel framework that leverages RL from Human Feedback (RLHF) to quantify human preferences, covering abstractness, and utilizes them to guide diffusion planning for zero-shot behavior customizing, covering mutability. AlignDiff can accurately match user-customized behaviors and efficiently switch from one to another. To build the framework, we first establish the multi-perspective human feedback datasets, which contain comparisons for the attributes of diverse behaviors, and then train an attribute strength model to predict quantified relative strengths. After relabeling behavioral datasets with relative strengths, we proceed to train an attribute-conditioned diffusion model, which serves as a planner with the attribute strength model as a director for preference aligning at the inference phase. We evaluate AlignDiff on various locomotion tasks and demonstrate its superior performance on preference matching, switching, and covering compared to other baselines. Its capability of completing unseen downstream tasks under human instructions also showcases the promising potential for human-AI collaboration. More visualization videos are released on <a class="link-external link-https" href="https://aligndiff.github.io/" rel="external noopener nofollow">this https URL</a>.

Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the issue of aligning diverse user preferences with agent behavior in Reinforcement Learning (RL). Specifically, the paper focuses on two challenges: 1. **Abstraction**: The abstract nature of human preferences makes it difficult to directly quantify preferences through manually designed reward functions. 2. **Variability**: Due to individual differences and changing preferences over time, it is necessary to flexibly adjust behavior to accommodate different preferences. To tackle these challenges, the paper proposes a new framework called AlignDiff, which leverages human feedback-based methods in Reinforcement Learning (RLHF) to guide diffusion planning by quantifying human preferences, thereby achieving zero-shot behavior customization. AlignDiff can accurately match user-customized behaviors and efficiently switch between different behaviors. Additionally, the paper demonstrates the superior performance of AlignDiff in various motion tasks, particularly in preference matching, switching, and coverage, and proves its great potential in completing unseen tasks under human guidance.

AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model

Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control

Diffusion Model Alignment Using Direct Preference Optimization

Regularized Conditional Diffusion Model for Multi-Task Preference Alignment

Forward KL Regularized Preference Optimization for Aligning Diffusion Policies

Adaptive Preference Scaling for Reinforcement Learning with Human Feedback

Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment

Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences

Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignment

Reinforcement Learning from Diverse Human Preferences

Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback

Enhancing Exploration with Diffusion Policies in Hybrid Off-Policy RL: Application to Non-Prehensile Manipulation

Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback

Aligning Diffusion Models with Noise-Conditioned Perception

Preference Aligned Diffusion Planner for Quadrupedal Locomotion Control

RLIF: Interactive Imitation Learning as Reinforcement Learning

Alignment of Diffusion Models: Fundamentals, Challenges, and Future

ATraDiff: Accelerating Online Reinforcement Learning with Imaginary Trajectories

Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient