Monocular Event-Based Vision for Obstacle Avoidance with a Quadrotor

Anish Bhattacharya,Marco Cannici,Nishanth Rao,Yuezhan Tao,Vijay Kumar,Nikolai Matni,Davide Scaramuzza

2024-11-06

Abstract:We present the first static-obstacle avoidance method for quadrotors using just an onboard, monocular event camera. Quadrotors are capable of fast and agile flight in cluttered environments when piloted manually, but vision-based autonomous flight in unknown environments is difficult in part due to the sensor limitations of traditional onboard cameras. Event cameras, however, promise nearly zero motion blur and high dynamic range, but produce a very large volume of events under significant ego-motion and further lack a continuous-time sensor model in simulation, making direct sim-to-real transfer not possible. By leveraging depth prediction as a pretext task in our learning framework, we can pre-train a reactive obstacle avoidance events-to-control policy with approximated, simulated events and then fine-tune the perception component with limited events-and-depth real-world data to achieve obstacle avoidance in indoor and outdoor settings. We demonstrate this across two quadrotor-event camera platforms in multiple settings and find, contrary to traditional vision-based works, that low speeds (1m/s) make the task harder and more prone to collisions, while high speeds (5m/s) result in better event-based depth estimation and avoidance. We also find that success rates in outdoor scenes can be significantly higher than in certain indoor scenes.

Robotics

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper aims to address the problem of static obstacle avoidance for quadrotor drones during autonomous flight in unknown environments. Specifically, the authors propose a method for static obstacle avoidance using only a monocular event camera. Traditional cameras face significant challenges in autonomous flight in complex environments due to low frame rates, limited dynamic range, and motion blur. In contrast, event cameras offer advantages such as low latency and high dynamic range. However, the event stream output from event cameras generates a large amount of data during high-speed motion and lacks a continuous-time sensor model, making the transition from simulation to real-world application difficult.

### Main Contributions

1. **First proposed method for static obstacle avoidance based on event streams**: This is the first method to achieve static obstacle avoidance using a monocular event camera.
2. **Successful few-shot transfer from simulation to real-world application**: By utilizing depth prediction as a pre-training task, the event stream-based strategy effectively transfers from simulation to real-world application.
3. **High-speed (5m/s) obstacle avoidance in real environments**: Demonstrated that the drone can successfully avoid trees at high speeds (5m/s) using purely onboard computation, with improved obstacle avoidance performance as speed increases.
4. **Open-source code and data**: Provided open-source code and data for simulation, data collection, training, and testing.

### Method Overview

1. **Simulation setup and data collection**: Used the Flightmare simulator to generate grayscale and depth images, and converted these images to event streams using the Vid2E tool.
2. **Event data generation and representation**: Batch processed the event stream into binary event masks (BEM) within time windows to reduce the impact of discontinuities.
3. **Learning framework**: The perception module (D(θ)) uses a U-Net architecture to predict depth from BEM, and the velocity prediction module (V(Φ)) generates obstacle avoidance velocity commands based on the depth prediction results.
4. **Few-shot transfer and cross-platform adaptation**: Collected real-world event and depth image pairs to fine-tune the perception module, adapting it to different scenes and platforms.

### Experimental Results

1. **Simulation experiments**: Conducted 100 rolling trials in a simulated forest environment, showing a 60% success rate on medium-length trajectories (10m) and 15% of trials without collisions on long trajectories (60m).
2. **Hardware experiments**: Conducted experiments in indoor and outdoor environments using different event camera platforms. Results showed that the event stream-based obstacle avoidance method performed better at high speeds (5m/s) compared to traditional vision methods. More events generated at higher speeds improved depth prediction quality, enhancing obstacle avoidance performance.

### Conclusion

This paper validates the feasibility of using only event cameras for static obstacle avoidance through simulation and real-world experiments. The study found that, unlike traditional vision methods, the event stream-based obstacle avoidance method performs better at high speeds because higher speeds generate more events, improving depth prediction quality. Additionally, the success rate in outdoor tests was higher than in indoor tests, possibly due to artificial light sources and background textures in indoor environments interfering with event pattern recognition.