Abstract:We focus on the problem of reliably training Reinforcement Learning (RL) models (agents) for stable low-level control in embedded systems and test our methods on a high-performance, custom-built quadrotor platform. A common but often under-studied problem in developing RL agents for continuous control is that the control policies developed are not always smooth. This lack of smoothness can be a major problem when learning controllers as it can result in control instability and hardware failure. Issues of noisy control are further accentuated when training RL agents in simulation due to simulators ultimately being imperfect representations of reality—what is known as the reality gap . To combat issues of instability in RL agents, we propose a systematic framework, REinforcement-based transferable Agents through Learning (RE+AL), for designing simulated training environments that preserve the quality of trained agents when transferred to real platforms. RE+AL is an evolution of the Neuroflight infrastructure detailed in technical reports prepared by members of our research group. Neuroflight is a state-of-the-art framework for training RL agents for low-level attitude control. RE+AL improves and completes Neuroflight by solving a number of important limitations that hindered the deployment of Neuroflight to real hardware. We benchmark RE+AL on the NF1 racing quadrotor developed as part of Neuroflight. We demonstrate that RE+AL significantly mitigates the previously observed issues of smoothness in RL agents. Additionally, RE+AL is shown to consistently train agents that are flight capable and with minimal degradation in controller quality upon transfer. RE+AL agents also learn to perform better than a tuned PID controller, with better tracking errors, smoother control, and reduced power consumption. To the best of our knowledge, RE+AL agents are the first RL-based controllers trained in simulation to outperform a well-tuned PID controller on a real-world controls problem that is solvable with classical control.

QPlane: An Open-Source Reinforcement Learning Toolkit for Autonomous Fixed Wing Aircraft Simulation

PyFlyt -- UAV Simulation Environments for Reinforcement Learning Research

How to Train Your Quadrotor: A Framework for Consistently Smooth and Responsive Flight Control via Reinforcement Learning

QuadSwarm: A Modular Multi-Quadrotor Simulator for Deep Reinforcement Learning with Direct Thrust Control

Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control

Reinforcement Learning for UAV Attitude Control

QuadSim: A Quadcopter Rotational Dynamics Simulation Framework For Reinforcement Learning Algorithms

Learning to Fly in Seconds

Interchangeable Reinforcement-Learning Flight Controller for Fixed-Wing UASs

Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

Drone Elevation Control Based on Python-Unity Integrated Framework for Reinforcement Learning Applications

Attitude Control of Highly Maneuverable Aircraft Using an Improved Q-learning

Data-Efficient Deep Reinforcement Learning for Attitude Control of Fixed-Wing UAVs: Field Experiments

A deep reinforcement learning control approach for high-performance aircraft

PLANRL: A Motion Planning and Imitation Learning Framework to Bootstrap Reinforcement Learning

Air Learning: A Deep Reinforcement Learning Gym for Autonomous Aerial Robot Visual Navigation

ProxFly: Robust Control for Close Proximity Quadcopter Flight via Residual Reinforcement Learning

A Survey on Reinforcement Learning in Aviation Applications

Teach-Repeat-Replan: A Complete and Robust System for Aggressive Flight in Complex Environments

Model-assisted Reinforcement Learning of a Quadrotor

Data-Driven Aircraft Modeling for Robust RL Control Synthesis with Flight Test Validation