Kalman meets Bellman: Improving Policy Evaluation through Value Tracking

Shirli Di-Castro Shashua,Shie Mannor

DOI: https://doi.org/10.48550/arXiv.2002.07171

2020-02-17

Abstract:Policy evaluation is a key process in Reinforcement Learning (RL). It assesses a given policy by estimating the corresponding value function. When using parameterized value functions, common approaches minimize the sum of squared Bellman temporal-difference errors and receive a point-estimate for the parameters. Kalman-based and Gaussian-processes based frameworks were suggested to evaluate the policy by treating the value as a random variable. These frameworks can learn uncertainties over the value parameters and exploit them for policy exploration. When adopting these frameworks to solve deep RL tasks, several limitations are revealed: excessive computations in each optimization step, difficulty with handling batches of samples which slows training and the effect of memory in stochastic environments which prevents off-policy learning. In this work, we discuss these limitations and propose to overcome them by an alternative general framework, based on the extended Kalman filter. We devise an optimization method, called Kalman Optimization for Value Approximation (KOVA) that can be incorporated as a policy evaluation component in policy optimization algorithms. KOVA minimizes a regularized objective function that concerns both parameter and noisy return uncertainties. We analyze the properties of KOVA and present its performance on deep RL control tasks.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve policy evaluation in Deep Reinforcement Learning (DRL). Specifically, the author points out some limitations of existing methods when dealing with deep - learning tasks, such as excessive computational complexity, difficulty in handling sample batches, and the memory - effect problem in stochastic environments. These problems impede the application of these methods in off - policy settings. To address these issues, the paper proposes a new framework - the Kalman Optimization for Value Approximation (KOVA) based on the Extended Kalman Filter (EKF), aiming to combine the advantages of the Kalman filter while overcoming the above - mentioned limitations. In particular, when dealing with nonlinear approximation functions such as Deep Neural Networks (DNNs), it can estimate the value function and model uncertainty more effectively. In this way, KOVA can provide more accurate policy evaluation in deep - reinforcement - learning tasks, promote policy exploration, and improve the overall performance of the algorithm.

Kalman meets Bellman: Improving Policy Evaluation through Value Tracking

Trust Region Value Optimization using Kalman Filtering

Inverse Policy Evaluation for Value-based Sequential Decision-making

General Policy Evaluation and Improvement by Learning to Identify Few But Crucial States

Iterative Amortized Policy Optimization

UVIP: Model-Free Approach to Evaluate Reinforcement Learning Algorithms

Kernel-Based Decentralized Policy Evaluation for Reinforcement Learning

Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models

Balancing Value Iteration and Policy Iteration for Discrete-Time Control.

Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization

Policy Optimization over General State and Action Spaces

Model-advantage and value-aware models for model-based reinforcement learning: bridging the gap in theory and practice

$K$-Nearest-Neighbor Resampling for Off-Policy Evaluation in Stochastic Control

Value Improved Actor Critic Algorithms

Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization

Kernel dynamic policy programming: Applicable reinforcement learning to robot systems with high dimensional states

Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone

A Variance Minimization Approach to Temporal-Difference Learning

Doubly Optimal Policy Evaluation for Reinforcement Learning

Least squares policy iteration with instrumental variables vs. direct policy search: comparison against optimal benchmarks using energy storage

Reinforcement learning in continuous time and space