Abstract:We focus on the problem of reliably training Reinforcement Learning (RL) models (agents) for stable low-level control in embedded systems and test our methods on a high-performance, custom-built quadrotor platform. A common but often under-studied problem in developing RL agents for continuous control is that the control policies developed are not always smooth. This lack of smoothness can be a major problem when learning controllers as it can result in control instability and hardware failure. Issues of noisy control are further accentuated when training RL agents in simulation due to simulators ultimately being imperfect representations of reality—what is known as the reality gap . To combat issues of instability in RL agents, we propose a systematic framework, REinforcement-based transferable Agents through Learning (RE+AL), for designing simulated training environments that preserve the quality of trained agents when transferred to real platforms. RE+AL is an evolution of the Neuroflight infrastructure detailed in technical reports prepared by members of our research group. Neuroflight is a state-of-the-art framework for training RL agents for low-level attitude control. RE+AL improves and completes Neuroflight by solving a number of important limitations that hindered the deployment of Neuroflight to real hardware. We benchmark RE+AL on the NF1 racing quadrotor developed as part of Neuroflight. We demonstrate that RE+AL significantly mitigates the previously observed issues of smoothness in RL agents. Additionally, RE+AL is shown to consistently train agents that are flight capable and with minimal degradation in controller quality upon transfer. RE+AL agents also learn to perform better than a tuned PID controller, with better tracking errors, smoother control, and reduced power consumption. To the best of our knowledge, RE+AL agents are the first RL-based controllers trained in simulation to outperform a well-tuned PID controller on a real-world controls problem that is solvable with classical control.

Evolutionary Reinforcement Learning: Hybrid Approach for Safety-Informed Fault-Tolerant Flight Control

Soft Actor-Critic Deep Reinforcement Learning for Fault Tolerant Flight Control

Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

Fault-tolerant Control for Unmanned Aerial Vehicle Using Deep Reinforcement Learning

Evolution-Guided Policy Gradient in Reinforcement Learning

How to Train Your Quadrotor: A Framework for Consistently Smooth and Responsive Flight Control via Reinforcement Learning

Enhanced Flight Envelope Protection: A Novel Reinforcement Learning Approach

Safe and Reliable Training of Learning-Based Aerospace Controllers

Reinforcement Learning-based Optimal Control and Software Rejuvenation for Safe and Efficient UAV Navigation

End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks

Runtime Safety Assurance Using Reinforcement Learning

A deep reinforcement learning control approach for high-performance aircraft

Reinforcement learning based closed‐loop reference model adaptive flight control system design

Verified Safe Reinforcement Learning for Neural Network Dynamic Models

Explainable and Safe Reinforcement Learning for Autonomous Air Mobility

Enhancing System-Level Safety in Mixed-Autonomy Platoon via Safe Reinforcement Learning

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

A Domain-Knowledge-Aided Deep Reinforcement Learning Approach for Flight Control Design

Evolutionary Reinforcement Learning: A Systematic Review and Future Directions

Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testing

Model-Based Safe Reinforcement Learning With Time-Varying Constraints: Applications to Intelligent Vehicles