Abstract:Inverse reinforcement learning (IRL) aims to explicitly infer an underlying reward function based on collected expert demonstrations. Considering that obtaining expert demonstrations can be costly, the focus of current IRL techniques is on learning a better-than-demonstrator policy using a reward function derived from sub-optimal demonstrations. However, existing IRL algorithms primarily tackle the challenge of trajectory ranking ambiguity when learning the reward function. They overlook the crucial role of considering the degree of difference between trajectories in terms of their returns, which is essential for further removing reward ambiguity. Additionally, it is important to note that the reward of a single transition is heavily influenced by the context information within the trajectory. To address these issues, we introduce the Distance-rank Aware Sequential Reward Learning (DRASRL) framework. Unlike existing approaches, DRASRL takes into account both the ranking of trajectories and the degrees of dissimilarity between them to collaboratively eliminate reward ambiguity when learning a sequence of contextually informed reward signals. Specifically, we leverage the distance between policies, from which the trajectories are generated, as a measure to quantify the degree of differences between traces. This distance-aware information is then used to infer embeddings in the representation space for reward learning, employing the contrastive learning technique. Meanwhile, we integrate the pairwise ranking loss function to incorporate ranking information into the latent features. Moreover, we resort to the Transformer architecture to capture the contextual dependencies within the trajectories in the latent space, leading to more accurate reward estimation. Through extensive experimentation, our DRASRL framework demonstrates significant performance improvements over previous SOTA methods.

Learning Long-Term Reward Redistribution via Randomized Return Decomposition

Episodic Return Decomposition by Difference of Implicitly Assigned Sub-Trajectory Reward

Towards Long-delayed Sparsity: Learning a Better Transformer Through Reward Redistribution.

Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach

Distributional Reward Decomposition for Reinforcement Learning

Reinforcement Learning from Bagged Reward

Inverse Reinforcement Learning with Multiple Ranked Experts

Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations

R3HF: Reward Redistribution for Enhancing Reinforcement Learning from Human Feedback

RUDDER: Return Decomposition for Delayed Rewards

Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learning

A Reward Optimization Method Based on Action Subrewards in Hierarchical Reinforcement Learning

Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning

Distributional Reinforcement Learning for Multi-Dimensional Reward Functions

Distributional Reinforcement Learning for Efficient Exploration

Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning

Rewarding Episodic Visitation Discrepancy for Exploration in Reinforcement Learning

DIFFER: Decomposing Individual Reward for Fair Experience Replay in Multi-Agent Reinforcement Learning

Reward Uncertainty for Exploration in Preference-based Reinforcement Learning

Auxiliary Reward Generation with Transition Distance Representation Learning

RD2: Reward Decomposition with Representation Disentanglement