Abstract:In many complex planning problems with factored, continuous state and action spaces such as Reservoir Control, Heating Ventilation and Air Conditioning (HVAC), and Navigation domains, it is difficult to obtain a model of the complex nonlinear dynamics that govern state evolution. However, the ubiquity of modern sensors allows us to collect large quantities of data from each of these complex systems and build accurate, nonlinear deep neural network models of their state transitions. But there remains one major problem for the task of control – how can we plan with deep network learned transition models without resorting to Monte Carlo Tree Search and other black-box transition model techniques that ignore model structure and do not easily extend to continuous domains? In this paper, we introduce two types of planning methods that can leverage deep neural network learned transition models: Hybrid Deep MILP Planner (HD-MILP-Plan) and Tensorflow Planner (TF-Plan). In HD-MILP-Plan, we make the critical observation that the Rectified Linear Unit (ReLU) transfer function for deep networks not only allows faster convergence of model learning, but also permits a direct compilation of the deep network transition model to a Mixed-Integer Linear Program (MILP) encoding. Further, we identify deep network specific optimizations for HD-MILP-Plan that improve performance over a base encoding and show that we can plan optimally with respect to the learned deep networks. In TF-Plan, we take advantage of the efficiency of auto-differentiation tools and GPU-based computation where we encode a subclass of purely continuous planning problems as Recurrent Neural Networks and directly optimize the actions through backpropagation. We compare both planners and show that TF-Plan is able to approximate the optimal plans found by HD-MILP-Plan in less computation time. Hence this article offers two novel planners for continuous state and action domains with learned deep neural net transition models: one optimal method (HD-MILP-Plan) and a scalable alternative for large-scale problems (TF-Plan).

Deep Recurrent Policy Networks for Planning under Partial Observability.

Spacecraft Attitude Maneuver Planning Based on Deep Reinforcement Learning under Complex Constraints

Partially Observable Planning and Learning for Systems with Non-Uniform Dynamics

A Deep Q Network for Robotic Planning from Image

A Scalable Model-Free Recurrent Neural Network Framework for Solving POMDPs

A Deep Bayesian Policy Reuse Approach Against Non-Stationary Agents.

On Improving Deep Reinforcement Learning for POMDPs

ODE-based Recurrent Model-free Reinforcement Learning for POMDPs

DM-DQN: Dueling Munchausen deep Q network for robot path planning

Efficient Policy Detecting and Reusing for Non-Stationarity in Markov Games.

Model-free Motion Planning of Autonomous Agents for Complex Tasks in Partially Observable Environments

Local Planning Strategy Based on Deep Reinforcement Learning Over Estimation Suppression

PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference

Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning

Making Better Decision by Directly Planning in Continuous Control

Provable Representation with Efficient Planning for Partial Observable Reinforcement Learning

Model-free reinforcement learning for motion planning of autonomous agents with complex tasks in partially observable environments

Dynamic Path Planning of Unknown Environment Based on Deep Reinforcement Learning

Scalable Planning with Deep Neural Network Learned Transition Models

Deep Recurrent Q-Learning for Partially Observable MDPs

Real-Time Motion Planning Of Kinematically Redundant Manipulators Using Recurrent Neural Networks