Abstract:State construction is important for learning in partially observable environments. A general purpose strategy for state construction is to learn the state update using a Recurrent Neural Network (RNN), which updates the internal state using the current internal state and the most recent observation. This internal state provides a summary of the observed sequence, to facilitate accurate predictions and decision-making. At the same time, specifying and training RNNs is notoriously tricky, particularly as the common strategy to approximate gradients back in time, called truncated Back-prop Through Time (BPTT), can be sensitive to the truncation window. Further, domain-expertise—which can usually help constrain the function class and so improve trainability—can be difficult to incorporate into complex recurrent units used within RNNs. In this work, we explore how to use multi-step predictions to constrain the RNN and incorporate prior knowledge. In particular, we revisit the idea of using predictions to construct state and ask: does constraining (parts of) the state to consist of predictions about the future improve RNN trainability? We formulate a novel RNN architecture, called a General Value Function Network (GVFN), where each internal state component corresponds to a prediction about the future represented as a value function. We first provide an objective for optimizing GVFNs, and derive several algorithms to optimize this objective. We then show that GVFNs are more robust to the truncation level, in many cases only requiring one-step gradient updates.

Asynchronous Value Iteration Network

Value Iteration Networks with Gated Summarization Module

Highway Value Iteration Networks

Scaling Value Iteration Networks to 5000 Layers for Extreme Long-Term Planning

Value Iteration Networks with Double Estimator for Planetary Rover Path Planning

Dynamic Value Iteration Networks for the Planning of Rapidly Changing UAV Swarms.

Capability Iteration Network for Robot Path Planning

Cooperative Motion Planning for Non-Holonomic Agents with Value Iteration Networks

General Value Function Networks

Iterative Amortized Policy Optimization

Dynamic sparse coding-based value estimation network for deep reinforcement learning

Asynchronous value iteration for markov decision processes with continuous state spaces

Deep Recurrent Policy Networks for Planning under Partial Observability.

Model-Assisted Reinforcement Learning with Adaptive Ensemble Value Expansion

Scaling Up and Stabilizing Differentiable Planning with Implicit Differentiation

Value Improved Actor Critic Algorithms

Vlearn: Off-Policy Learning with Efficient State-Value Function Estimation

AVD-Net: Attention Value Decomposition Network for Deep Multi-Agent Reinforcement Learning

A Neighborhood-Based Value Iteration Algorithm For Pomdp Problems

Continuous Value Assignment: A Doubly Robust Data Augmentation for Off-Policy Learning

Deflated Dynamics Value Iteration