An Open-source Hardware/Software Architecture and Supporting Simulation Environment to Perform Human FPV Flight Demonstrations for Unmanned Aerial Vehicle Autonomy

Haosong Xiao,Prajit Krisshnakumar,Jagadeswara P K V Pothuri,Puru Soni,Eric Butcher,Souma Chowdhury
2024-09-06
Abstract:Small multi-rotor unmanned aerial vehicles (UAVs), mainly quadcopters, are nowadays ubiquitous in research on aerial autonomy, including serving as scaled-down models for much larger aircraft such as vertical-take-off-and-lift vehicles for urban air mobility. Among the various research use cases, first-person-view RC flight experiments allow for collecting data on how human pilots fly such aircraft, which could then be used to compare, contrast, validate, or train autonomous flight agents. While this could be uniquely beneficial, especially for studying UAV operation in contextually complex and safety-critical environments such as in human-UAV shared spaces, the lack of inexpensive and open-source hardware/software platforms that offer this capability along with low-level access to the underlying control software and data remains limited. To address this gap and significantly reduce barriers to human-guided autonomy research with UAVs, this paper presents an open-source software architecture implemented with an inexpensive in-house built quadcopter platform based on the F450 Quadcopter Frame. This setup uses two cameras to provide a dual-view FPV and an open-source flight controller, Pixhawk. The underlying software architecture, developed using the Python-based Kivy library, allows logging telemetry, GPS, control inputs, and camera frame data in a synchronized manner on the ground station computer. Since costs (time) and weather constraints typically limit numbers of physical outdoor flight experiments, this paper also presents a unique AirSim/Unreal Engine based simulation environment and graphical user interface aka digital twin, that provides a Hardware In The Loop setup via the Pixhawk flight controller. We demonstrate the usability and reliability of the overall framework through a set of diverse physical FPV flight experiments and corresponding flight tests in the digital twin.
Robotics
What problem does this paper attempt to address?
The paper aims to address the following issues: 1. **Lack of low-cost and open-source hardware/software platforms**: Currently, there is a lack of low-cost and fully open-source hardware/software platforms on the market that can provide first-person view (FPV) flight experiment capabilities. Such platforms would allow researchers to access the underlying control software and data. 2. **Data collection and research on autonomous flight agents**: FPV flight experiments can collect operational data from human pilots, which can be used to compare, contrast, validate, or train (through imitation learning) autonomous flight agents. This capability is particularly beneficial when operating drones in complex and high-safety-requirement environments. 3. **Promoting research on human-guided drone autonomy**: To significantly reduce the barriers in drone autonomy research, it is necessary to develop an open-source software architecture that can be implemented on a low-cost, homemade quadcopter platform. 4. **Physical experiment limitations**: Due to cost and weather conditions, the number of actual outdoor flight experiments is greatly limited. Therefore, a unique simulation environment and graphical user interface (GUI) based on AirSim/Unreal Engine are also needed to provide hardware-in-the-loop (HITL) settings. In summary, the main goal of this paper is to introduce a cost-effective, easily accessible open-source framework with FPV capabilities and to develop a digital twin platform with HITL capabilities to provide an alternative for generating real, high-fidelity data when physical experiments are expensive or difficult to conduct frequently.