Abstract:In this study, we investigate the DIstribution Correction Estimation (DICE) methods, an important line of work in offline reinforcement learning (RL) and imitation learning (IL). DICE-based methods impose state-action-level behavior constraint, which is an ideal choice for offline learning. However, they typically perform much worse than current state-of-the-art (SOTA) methods that solely use action-level behavior constraint. After revisiting DICE-based methods, we find there exist two gradient terms when learning the value function using true-gradient update: forward gradient (taken on the current state) and backward gradient (taken on the next state). Using forward gradient bears a large similarity to many offline RL methods, and thus can be regarded as applying action-level constraint. However, directly adding the backward gradient may degenerate or cancel out its effect if these two gradients have conflicting directions. To resolve this issue, we propose a simple yet effective modification that projects the backward gradient onto the normal plane of the forward gradient, resulting in an orthogonal-gradient update, a new learning rule for DICE-based methods. We conduct thorough theoretical analyses and find that the projected backward gradient brings state-level behavior regularization, which reveals the mystery of DICE-based methods: the value learning objective does try to impose state-action-level constraint, but needs to be used in a corrected way. Through toy examples and extensive experiments on complex offline RL and IL tasks, we demonstrate that DICE-based methods using orthogonal-gradient updates (O-DICE) achieve SOTA performance and great robustness.

Greedy Criterion in Orthogonal Greedy Learning.

Greedy metrics in orthogonal greedy learning.

Learning and Approximation Capabilities of Orthogonal Super Greedy Algorithm

Optimal Convergence Rates for the Orthogonal Greedy Algorithm

Greedy Learning to Optimize with Convergence Guarantees

Orthogonal Super Greedy Learning for Sparse Feedforward Neural Networks

Learning Capability of the Truncated Greedy Algorithm

Accelerated Stochastic Greedy Coordinate Descent by Soft Thresholding Projection Onto Simplex

Adaptive, Doubly Optimal No-Regret Learning in Strongly Monotone and Exp-Concave Games with Gradient Feedback

ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update

Greedy coordinate descent from the view of $\ell_1$-norm gradient descent

Orthogonal Gradient Descent for Continual Learning

The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization

Finite-Time Error Bounds for Greedy-GQ

Efficient and provably convergent randomized greedy algorithms for neural network optimization

A Generalized Alternating Method for Bilevel Learning under the Polyak-Łojasiewicz Condition

Optimization-Derived Learning with Essential Convergence Analysis of Training and Hyper-training

Super greedy type algorithms

A New Adaptive Gradient Method with Gradient Decomposition

Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization

Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold