Abstract:A major challenge in robotics is to design robust policies which enable complex and agile behaviors in the real world. On one end of the spectrum, we have model-free reinforcement learning (MFRL), which is incredibly flexible and general but often results in brittle policies. In contrast, model predictive control (MPC) continually re-plans at each time step to remain robust to perturbations and model inaccuracies. However, despite its real-world successes, MPC often under-performs the optimal strategy. This is due to model quality, myopic behavior from short planning horizons, and approximations due to computational constraints. And even with a perfect model and enough compute, MPC can get stuck in bad local optima, depending heavily on the quality of the optimization algorithm. To this end, we propose Deep Model Predictive Optimization (DMPO), which learns the inner-loop of an MPC optimization algorithm directly via experience, specifically tailored to the needs of the control problem. We evaluate DMPO on a real quadrotor agile trajectory tracking task, on which it improves performance over a baseline MPC algorithm for a given computational budget. It can outperform the best MPC algorithm by up to 27% with fewer samples and an end-to-end policy trained with MFRL by 19%. Moreover, because DMPO requires fewer samples, it can also achieve these benefits with 4.3X less memory. When we subject the quadrotor to turbulent wind fields with an attached drag plate, DMPO can adapt zero-shot while still outperforming all baselines. Additional results can be found at <a class="link-external link-https" href="https://tinyurl.com/mr2ywmnw" rel="external noopener nofollow">this https URL</a>.

Comparison of Deep Reinforcement Learning and Model Predictive Control for Real-Time Depth Optimization of a Lifting Surface Controlled Ocean Current Turbine

Robust Dynamic Positioning of Autonomous Surface Vessels with Tube-Based Model Predictive Control

A Model Predictive Control Approach for USV Autonomous Cruising Via Disturbance Learning

Deep Model Predictive Optimization

Integrated Path Planning and Tracking Control of Marine Current Turbine in Uncertain Ocean Environments

Environmental-Sensing and adaptive optimization of wave energy converter based on deep reinforcement learning and computational fluid dynamics

Reinforcement Learning Ship Autopilot: Sample efficient and Model Predictive Control-based Approach

Real-time model predictive control study of run-of-river hydropower plants with data-driven and physics-based coupled model

A Time and Place to Land: Online Learning-Based Distributed MPC for Multirotor Landing on Surface Vessel in Waves

Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control

Jointly improving energy efficiency and smoothing power oscillations of integrated offshore wind and photovoltaic power: a deep reinforcement learning approach

Joint Optimization of Power Generation and Voyage Scheduling in Ship Power System Based on Operating Scene Clustering and Multi-Task Deep Reinforcement Learning

Control of a Wave Energy Converter Using Model-Free Deep Reinforcement Learning

MPC-based Reinforcement Learning for a Simplified Freight Mission of Autonomous Surface Vehicles

Dynamic joint optimization of power generation and voyage scheduling in ship power system based on deep reinforcement learning

Dynamic Obstacle Avoidance for USVs Using Cross-Domain Deep Reinforcement Learning and Neural Network Model Predictive Controller

Deep Reinforcement Learning Based Controller for Active Heave Compensation

DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigation

Data-centric predictive control with tuna swarm optimization-backpropagation neural networks for enhanced wind turbine performance

Deep Value Model Predictive Control

Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms