Real-Time Neuromorphic Navigation: Integrating Event-Based Vision and Physics-Driven Planning on a Parrot Bebop2 Quadrotor

Amogh Joshi,Sourav Sanyal,Kaushik Roy

2024-07-01

Abstract:In autonomous aerial navigation, real-time and energy-efficient obstacle avoidance remains a significant challenge, especially in dynamic and complex indoor environments. This work presents a novel integration of neuromorphic event cameras with physics-driven planning algorithms implemented on a Parrot Bebop2 quadrotor. Neuromorphic event cameras, characterized by their high dynamic range and low latency, offer significant advantages over traditional frame-based systems, particularly in poor lighting conditions or during high-speed maneuvers. We use a DVS camera with a shallow Spiking Neural Network (SNN) for event-based object detection of a moving ring in real-time in an indoor lab. Further, we enhance drone control with physics-guided empirical knowledge inside a neural network training mechanism, to predict energy-efficient flight paths to fly through the moving ring. This integration results in a real-time, low-latency navigation system capable of dynamically responding to environmental changes while minimizing energy consumption. We detail our hardware setup, control loop, and modifications necessary for real-world applications, including the challenges of sensor integration without burdening the flight capabilities. Experimental results demonstrate the effectiveness of our approach in achieving robust, collision-free, and energy-efficient flight paths, showcasing the potential of neuromorphic vision and physics-driven planning in enhancing autonomous navigation systems.

Robotics

What problem does this paper attempt to address?

The paper primarily focuses on addressing the problem of real-time and energy-efficient obstacle avoidance navigation for autonomous flying vehicles (specifically quadrotor drones) in dynamic and complex indoor environments. Specifically, the authors propose a novel approach that combines event-based neuromorphic vision with a physics-driven planning algorithm and implements this integration on the Parrot Bebop2 quadrotor drone.

The core technical points include:
1. **Event-based vision system**: Utilizing a neuromorphic event camera (DVS), which features a higher dynamic range and lower latency compared to traditional frame-based cameras, especially under poor lighting conditions or during high-speed maneuvers.
2. **Shallow Spiking Neural Network (SNN)**: Used for real-time detection of moving ring-shaped objects, in conjunction with the DVS camera.
3. **Integration of physics-guided empirical knowledge into the neural network training mechanism**: This approach predicts energy-efficient flight paths, enabling the drone to navigate through moving ring obstacles.
4. **Hardware and software integration**: Detailed descriptions of the hardware setup, control loop, and necessary modifications for practical application, including how to overcome the impact of sensor integration on flight capabilities.

Experimental results indicate that this method effectively achieves stable, collision-free, and low-energy flight paths, demonstrating the potential of neuromorphic vision and physics-driven planning in enhancing autonomous navigation systems. However, there is an increase in energy consumption from simulation to real-world application, reflecting the challenges posed by environmental factors and mechanical limitations in the real world.