Dynamic obstacle avoidance for quadrotors with event cameras

Davide Falanga,Kevin Kleber,Davide Scaramuzza

DOI: https://doi.org/10.1126/scirobotics.aaz9712

IF: 25

2020-03-25

Science Robotics

Abstract:Micro-aerial vehicles dodge fast-moving objects using only onboard sensing and computation.

robotics

What problem does this paper attempt to address?

Monocular Event-Based Vision for Obstacle Avoidance with a Quadrotor

Anish Bhattacharya,Marco Cannici,Nishanth Rao,Yuezhan Tao,Vijay Kumar,Nikolai Matni,Davide Scaramuzza

2024-11-06

Abstract:We present the first static-obstacle avoidance method for quadrotors using just an onboard, monocular event camera. Quadrotors are capable of fast and agile flight in cluttered environments when piloted manually, but vision-based autonomous flight in unknown environments is difficult in part due to the sensor limitations of traditional onboard cameras. Event cameras, however, promise nearly zero motion blur and high dynamic range, but produce a very large volume of events under significant ego-motion and further lack a continuous-time sensor model in simulation, making direct sim-to-real transfer not possible. By leveraging depth prediction as a pretext task in our learning framework, we can pre-train a reactive obstacle avoidance events-to-control policy with approximated, simulated events and then fine-tune the perception component with limited events-and-depth real-world data to achieve obstacle avoidance in indoor and outdoor settings. We demonstrate this across two quadrotor-event camera platforms in multiple settings and find, contrary to traditional vision-based works, that low speeds (1m/s) make the task harder and more prone to collisions, while high speeds (5m/s) result in better event-based depth estimation and avoidance. We also find that success rates in outdoor scenes can be significantly higher than in certain indoor scenes.

Robotics
Autonomous Flights in Dynamic Environments with Onboard Vision

Yingjian Wang,Jialin Ji,Qianhao Wang,Chao Xu,Fei Gao

DOI: https://doi.org/10.1109/IROS51168.2021.9636117

2021-01-01

Abstract:In this paper, we introduce a complete system for autonomous flight of quadrotors in dynamic environments with onboard sensing. Extended from existing work, we develop an occlusion-aware dynamic perception method based on depth images, which classifies obstacles as dynamic and static. For representing generic dynamic environment, we model dynamic objects with moving ellipsoids and fuse static ones into an occupancy grid map. To achieve dynamic avoidance, we design a planning method composed of modified kinodynamic path searching and gradient-based optimization. The method leverages manually constructed gradients without maintaining a signed distance field (SDF), making the planning procedure finished in milliseconds. We integrate the above methods into a customized quadrotor system and thoroughly test it in real-world experiments, verifying its effective collision avoidance in dynamic environments.
Robust Vision-based Obstacle Avoidance for Micro Aerial Vehicles in Dynamic Environments

Jiahao Lin,Hai Zhu,Javier Alonso-Mora

DOI: https://doi.org/10.48550/arXiv.2002.04920

2020-02-13

Abstract:In this paper, we present an on-board vision-based approach for avoidance of moving obstacles in dynamic environments. Our approach relies on an efficient obstacle detection and tracking algorithm based on depth image pairs, which provides the estimated position, velocity and size of the obstacles. Robust collision avoidance is achieved by formulating a chance-constrained model predictive controller (CC-MPC) to ensure that the collision probability between the micro aerial vehicle (MAV) and each moving obstacle is below a specified threshold. The method takes into account MAV dynamics, state estimation and obstacle sensing uncertainties. The proposed approach is implemented on a quadrotor equipped with a stereo camera and is tested in a variety of environments, showing effective on-line collision avoidance of moving obstacles.

Robotics
Fusing Stereopsis & Corner Sparse Optical Flow for Real-time Obstacle Avoidance of Unmanned Aerial Vehicle

Shihao Ding,Xiaoxia Li,Shuoquan Wei,XiangChun Kong,Wenjun Yan

DOI: https://doi.org/10.1109/codit.2019.8820663

2019-01-01

Abstract:The autonomous and maneuverable flight for an Unmanned Aerial Vehicle (UAV) in a complicated environment is an urgent problem to be addressed. This paper presents a real-time obstacle avoidance solution to autonomous flight for UAV systems. A binocular stereo matching model and an enhanced Pyramidal Lucas Kanade Optical Flow (PLKOF) algorithm were built for fast obstacle detection. The stereopsis fused with corner sparse optical flow is presented to an obstacle avoidance algorithm. The proposed obstacle avoidance solution is efficiently running on the DJI M100 equipped with a multiple sensor system Guidance to detect obstacle features and produce efficient motions at a frequency of 5 Hz in a multi-task condition. The solution is evaluated through rigorous comparative experiments and the results demonstrate its effectiveness obviously.
Avoiding dynamic small obstacles with onboard sensing and computating on aerial robots

Fanze Kong,Wei Xu,Fu Zhang

DOI: https://doi.org/10.48550/arXiv.2103.00406

2021-02-28

Abstract:In practical applications, autonomous quadrotors are still facing significant challenges, such as the detection and avoidance of very small and even dynamic obstacles (e.g., tree branches, power lines). In this paper, we propose a compact, integrated, and fully autonomous quadrotor system, which can fly safely in cluttered environments while avoiding dynamic small obstacles. Our quadrotor platform is equipped with a forward-looking three-dimensional (3D) light detection and ranging (lidar) sensor to perceive the environment and an onboard embedded computer to perform all the estimation, mapping, and planning tasks. Specifically, the computer estimates the current pose of the UAV, maintains a local map (time-accumulated point clouds KD-Trees), and computes a safe trajectory using kinodynamic A* search to the goal point. The whole perception and planning system can run onboard at 50Hz with careful optimization. Various indoor and outdoor experiments show that the system can avoid dynamic small obstacles (down to 20mm diameter bar) while flying at 2m/s in cluttered environments. Our codes and hardware design are open-sourced on Github.

Robotics
A real-time dynamic obstacle tracking and mapping system for UAV navigation and collision avoidance with an RGB-D camera

Zhefan Xu,Xiaoyang Zhan,Baihan Chen,Yumeng Xiu,Chenhao Yang,Kenji Shimada

DOI: https://doi.org/10.48550/arXiv.2209.08258

IF: 3.7

2022-09-17

Robotics

Abstract:The real-time dynamic environment perception has become vital for autonomous robots in crowded spaces. Although the popular voxel-based mapping methods can efficiently represent 3D obstacles with arbitrarily complex shapes, they can hardly distinguish between static and dynamic obstacles, leading to the limited performance of obstacle avoidance. While plenty of sophisticated learning-based dynamic obstacle detection algorithms exist in autonomous driving, the quadcopter's limited computation resources cannot achieve real-time performance using those approaches. To address these issues, we propose a real-time dynamic obstacle tracking and mapping system for quadcopter obstacle avoidance using an RGB-D camera. The proposed system first utilizes a depth image with an occupancy voxel map to generate potential dynamic obstacle regions as proposals. With the obstacle region proposals, the Kalman filter and our continuity filter are applied to track each dynamic obstacle. Finally, the environment-aware trajectory prediction method is proposed based on the Markov chain using the states of tracked dynamic obstacles. We implemented the proposed system with our custom quadcopter and navigation planner. The simulation and physical experiments show that our methods can successfully track and represent obstacles in dynamic environments in real-time and safely avoid obstacles.
Monocular Vision-Based Obstacle Detection/Avoidance For Unmanned Aerial Vehicles

Abdulla Al-Kaff,Qinggang Meng,David Martin,Arturo De La Escalera,Jose Maria Armingol

DOI: https://doi.org/10.1109/IVS.2016.7535370

2016-01-01

Abstract:Robust real-time obstacle detection/avoidance is a challenging problem especially for micro and small aerial vehicles due to the limited number of the on-board sensors due to the battery constraint and low payload. Usually lightweight sensors such as CMOS camera are the best choice comparing with laser or radar sensors. For real-time applications, most studies focus on using stereo cameras to reconstruct a 3D model of the obstacles or to estimate their depth. Instead, in this paper, a method that mimics the human behavior of detecting the state of the approaching obstacles using single camera is proposed. During the flight, this method is able to detect the changes of the size area of the obstacles. First, the method detects the feature points of the obstacles, and then extracts the obstacles that has probability of getting close. In addition, by comparing the changes in the area ratios of the obstacle in the image sequence, the method can decide if it is obstacle or not. Finally, by estimating the obstacle 2D position in the image and combining with the tracked waypoints, the UAV can take the action of avoidance.
Free as a Bird: Event-based Dynamic Sense-and-Avoid for Ornithopter Robot Flight

J.P. Rodríguez-Gómez,R. Tapia,M.M. Guzmán,J.R. Martínez-de Dios,A. Ollero

DOI: https://doi.org/10.1109/LRA.2022.3153904

2023-10-16

Abstract:Autonomous flight of flapping-wing robots is a major challenge for robot perception. Most of the previous sense-and-avoid works have studied the problem of obstacle avoidance for flapping-wing robots considering only static obstacles. This paper presents a fully onboard dynamic sense-and-avoid scheme for large-scale ornithopters using event cameras. These sensors trigger pixel information due to changes of illumination in the scene such as those produced by dynamic objects. The method performs event-by-event processing in low-cost hardware such as those onboard small aerial vehicles. The proposed scheme detects obstacles and evaluates possible collisions with the robot body. The onboard controller actuates over the horizontal and vertical tail deflections to execute the avoidance maneuver. The scheme is validated in both indoor and outdoor scenarios using obstacles of different shapes and sizes. To the best of the authors' knowledge, this is the first event-based method for dynamic obstacle avoidance in a flapping-wing robot.

Robotics
3-D Reciprocal Collision Avoidance on Physical Quadrotor Helicopters with On-Board Sensing for Relative Positioning

Parker Conroy,Daman Bareiss,Matt Beall,Jur van den Berg

DOI: https://doi.org/10.48550/arXiv.1411.3794

IF: 3.7

2014-11-14

Robotics

Abstract:In this paper, we present an implementation of 3-D reciprocal collision avoidance on real quadrotor helicopters where each quadrotor senses the relative position and velocity of other quadrotors using an on-board camera. We show that using our approach, quadrotors are able to successfully avoid pairwise collisions in GPS and motion-capture denied environments, without communication between the quadrotors, and even when human operators deliberately attempt to induce collisions. To our knowledge, this is the first time that reciprocal collision avoidance has been successfully implemented on real robots where each agent independently observes the others using on-board sensors. We theoretically analyze the response of the collision-avoidance algorithm to the violated assumptions by the use of real robots. We quantitatively analyze our experimental results. A particularly striking observation is that at times the quadrotors exhibit "reciprocal dance" behavior, which is also observed when humans move past each other in constrained environments. This seems to be the result of sensing uncertainty, which causes both robots involved to have a different belief about the relative positions and velocities and, as a result, choose the same side on which to pass.
Obstacle Avoidance of Monocular Quadrotors with Depth Estimation

Liu Qi,Zhang Zhenghong,Xiong Mingkang,Xiong Huilin

DOI: https://doi.org/10.1007/978-981-16-9492-9_313

2022-01-01

Abstract:Obstacle avoidance based on a monocular camera is still a challenging task because of the deficiency of 3D information for a monocular camera of the quadrotor. In this paper, we present an obstacle avoidance system for quadrotor based on monocular vision and depth estimation. The training data derives from the flight video of the front camera. Based on the depth map, we represent the obstacles in an expansion space, and then apply our motion planning algorithm to avoid the collisions. Real-world experiments verify its high-performance.
FAST-Dynamic-Vision: Detection and Tracking Dynamic Objects with Event and Depth Sensing

Botao He,Haojia Li,Siyuan Wu,Dong Wang,Zhiwei Zhang,Qianli Dong,Chao Xu,Fei Gao

DOI: https://doi.org/10.1109/iros51168.2021.9636448

2021-09-27

Abstract:The development of aerial autonomy has enabled aerial robots to fly agilely in complex environments. However, dodging fast-moving objects in flight remains a challenge, limiting the further application of unmanned aerial vehicles (UAVs). The bottleneck of solving this problem is the accurate perception of rapid dynamic objects. Recently, event cameras have shown great potential in solving this problem. This paper presents a complete perception system including ego-motion compensation, object detection, and trajectory prediction for fast-moving dynamic objects with low latency and high precision. Firstly, we propose an accurate ego-motion compensation algorithm by considering both rotational and translational motion for more robust object detection. Then, for dynamic object detection, an event camera-based efficient regression algorithm is designed. Finally, we propose an optimization-based approach that asynchronously fuses event and depth cameras for trajectory prediction. Extensive real-world experiments and benchmarks are performed to validate our framework. Moreover, our code will be released to benefit related researches.
Obstacle Avoidance Using Stereo Camera

Akkas Uddin Haque,Ashkan Nejadpak

DOI: https://doi.org/10.48550/arXiv.1705.04114

2017-05-17

Abstract:In this paper we present a novel method for obstacle avoidance using the stereo camera. The conventional obstacle avoidance methods and their limitations are discussed. A new algorithm is developed for the real-time obstacle avoidance which responds faster to unexpected obstacles. In this approach the depth map is divided into optimized number of regions and the minimum depth at each section is assigned as the depth of that region. A fuzzy controller is designed to create the drive commands for the robot/quadcopter. The system was tested on multiple paths with different obstacles and the results demonstrated the high accuracy of the developed system.

Computer Vision and Pattern Recognition
Risk-aware Trajectory Sampling for Quadrotor Obstacle Avoidance in Dynamic Environments

Gang Chen,Peng Peng,Peihan Zhang,Wei Dong

DOI: https://doi.org/10.48550/arXiv.2201.06645

2022-02-15

Abstract:Obstacle avoidance of quadrotors in dynamic environments is still a very open problem. Current works commonly leverage traditional static maps to represent static obstacles and the detection and tracking of moving objects (DATMO) method to model dynamic obstacles separately. The detection module requires pre-training, and the dynamic obstacles can only be modeled with certain shapes, such as cylinders or ellipsoids. This work utilizes the dual-structure particle-based (DSP) dynamic occupancy map to represent the arbitrary-shaped static obstacles and dynamic obstacles simultaneously, and proposes an efficient risk-aware sampling-based local trajectory planner to realize safe flights in this map. The trajectory is planned by sampling motion primitives generated in the state space. Each motion primitive is divided into two phases: a short-term phase with a strict risk limitation and a relatively long-term phase designed to avoid high-risk regions. The risk is evaluated with the predicted particle-form future occupancy status, considering the time dimension. With an approach to split from and merge to an arbitrary global trajectory, the planner can also be used in the tasks with preplanned global trajectories. Comparison experiments show that the obstacle avoidance system composed of the DSP map and our planner performs the best in dynamic environments. In real-world tests, our quadrotor reaches a speed of 6 m/s with the motion capture system and 2.5 m/s with everything running on a low-price single-board computer.

Robotics
Taming Event Cameras with Bio-Inspired Architecture and Algorithm: A Case for Drone Obstacle Avoidance

Yishujie Zhao,Danyang Li,Longfei Shangguan,Zheng Yang,Hao Cao,Yunhao Liu,Jingao Xu

DOI: https://doi.org/10.1145/3570361.3613269

2023-10-02

Abstract:Fast and accurate obstacle avoidance is crucial to drone safety. Yet existing on-board sensor modules such as frame cameras and radars are ill-suited for doing so due to their low temporal resolution or limited field of view. This paper presents BioDrone, a new design paradigm for drone obstacle avoidance using stereo event cameras. At the heart of BioDrone is two simple yet effective system design inspired by the mammalian visual system, namely, a chiasm-inspired signal processing pipeline for fast event filtering and obstacle detection, and a lateral geniculate nucleus (LGN)-inspired event matching algorithm for accurate obstacle localization. To make BioDrone a practical solution, we further take significant engineering efforts to deploy the software stack on FPGA through software and hardware co-design. The performance comparison with two state-of-the-art event-based obstacle avoidance systems shows BioDrone achieves a consistently high obstacle detection rate of 96.1%. The average localization error of BioDrone is 6.8cm with a 4.7ms latency, outperforming both baselines by over 40%.

Engineering,Computer Science
Detecting and avoiding frontal obstacles from monocular camera for micro unmanned aerial vehicles

Mohamed Elawady,Ibrahim Sadek,Hiliwi Kidane

DOI: https://doi.org/10.48550/arXiv.1603.09422

2016-04-03

Abstract:In literature, several approaches are trying to make the UAVs fly autonomously i.e., by extracting perspective cues such as straight lines. However, it is only available in well-defined human made environments, in addition to many other cues which require enough texture information. Our main target is to detect and avoid frontal obstacles from a monocular camera using a quad rotor <a class="link-external link-http" href="http://Ar.Drone" rel="external noopener nofollow">this http URL</a> 2 by exploiting optical flow as a motion parallax, the drone is permitted to fly at a speed of 1 m/s and an altitude ranging from 1 to 4 meters above the ground level. In general, detecting and avoiding frontal obstacle is a quite challenging problem because optical flow has some limitation which should be taken into account i.e. lighting conditions and aperture problem.

Robotics
Real-time Quadrotor Navigation Through Planning in Depth Space in Unstructured Environments

Shakeeb Ahmad,Rafael Fierro

DOI: https://doi.org/10.48550/arXiv.2010.09098

IF: 3.7

2020-10-18

Robotics

Abstract:This paper addresses the problem of real-time vision-based autonomous obstacle avoidance in unstructured environments for quadrotor UAVs. We assume that our UAV is equipped with a forward facing stereo camera as the only sensor to perceive the world around it. Moreover, all the computations are performed onboard. Feasible trajectory generation in this kind of problems requires rapid collision checks along with efficient planning algorithms. We propose a trajectory generation approach in the depth image space, which refers to the environment information as depicted by the depth images. In order to predict the collision in a look ahead robot trajectory, we create depth images from the sequence of robot poses along the path. We compare these images with the depth images of the actual world sensed through the forward facing stereo camera. We aim at generating fuel optimal trajectories inside the depth image space. In case of a predicted collision, a switching strategy is used to aggressively deviate the quadrotor away from the obstacle. For this purpose we use two closed loop motion primitives based on Linear Quadratic Regulator (LQR) objective functions. The proposed approach is validated through simulation and hardware experiments.
EVReflex: Dense Time-to-Impact Prediction for Event-based Obstacle Avoidance

Celyn Walters,Simon Hadfield

DOI: https://doi.org/10.1109/IROS51168.2021.9636327

2021-09-01

Abstract:The broad scope of obstacle avoidance has led to many kinds of computer vision-based approaches. Despite its popularity, it is not a solved problem. Traditional computer vision techniques using cameras and depth sensors often focus on static scenes, or rely on priors about the obstacles. Recent developments in bio-inspired sensors present event cameras as a compelling choice for dynamic scenes. Although these sensors have many advantages over their frame-based counterparts, such as high dynamic range and temporal resolution, event-based perception has largely remained in 2D. This often leads to solutions reliant on heuristics and specific to a particular task. We show that the fusion of events and depth overcomes the failure cases of each individual modality when performing obstacle avoidance. Our proposed approach unifies event camera and lidar streams to estimate metric time-to-impact without prior knowledge of the scene geometry or obstacles. In addition, we release an extensive event-based dataset with six visual streams spanning over 700 scanned scenes.

Computer Vision and Pattern Recognition,Robotics
Monocular Vision-Based Dynamic Moving Obstacles Detection and Avoidance

Wilbert G. Aguilar,Leandro Álvarez,Santiago Grijalva,Israel Rojas

DOI: https://doi.org/10.1007/978-3-030-27541-9_32

2019-01-01

Abstract:In this paper, we proposed an UAV system for obstacle avoidance depending of a ground station processing based on monocular vision. To accomplish detection, tracking, proximity estimation and dynamic obstacles avoidance that are approaching the UAV, a series of methods and techniques are implemented. To detect movement, frame differentiation was applied to consecutive frames, once the moving object is detected, we detect Shi-Tomasi feature points to track the object using optical flow method Lucas-Kanade. To make possible proximity estimation, a linear regression method based on the area covered by the object was used. A fuzzy logic controller was designed to avoid the moving object, we considered the approach rate and the area of the object to control UAV’s position in relation to the object as fuzzy inputs.
Monocular Snapshot-based Sensing and Control of Hover, Takeoff, and Landing for a Low-cost Quadrotor

Ping Li,Matthew A. Garratt,Andrew J. Lambert

DOI: https://doi.org/10.1002/rob.21574

IF: 6.385

2015-01-01

Journal of Field Robotics

Abstract:AbstractAutonomous control of a low-cost micro air vehicle MAV is considered, using a single camera and inertial measurement unit. Integration of loom is used to determine the approximate height during takeoff and landing, and to provide an initial height for hover. Contrary to most optic-flow-based approaches that lack positional feedback, a "snapshot" paradigm is utilized in this paper that takes and stores an image of the ground under the MAV as a reference image, and the position of the vehicle can then be estimated from the transformation of the following images with respect to the reference image so that long-term drift can be contained. In this new work, the implementation of the snapshot algorithm is considered with a focus on real-time performance on a slow processor and the difficulties posed by illumination change and external disturbance. We also extend earlier work with extensive flight experiments conducted in various situations to evaluate the closed-loop performance of the proposed algorithm.
Monocular Depth Estimation for UAV Obstacle Avoidance

Zhenghong Zhang,Mingkang Xiong,Huilin Xiong

DOI: https://doi.org/10.1109/cciot48581.2019.8980350

2019-01-01

Abstract:In this paper, we present a novel method for obstacle avoidance of a quadrotor equiped with a single front camera. The proposed method is composed of three parts: depth estimation, obstacle detection, and obstacle avoidance controll. We use convolutional neural networks (CNN) to estimate depth from RGB image. Then the depth image is fed into the obstacle avoidance system, in which proposed control algorithm steers the quadrotor to fly away from obstacles, and after that, continue towards the destination. We conduct a lot of experiments, either in virtual environment with a simulated drone, or in real world with a quadrotor Parrot Bebop2, to verify the effectiveness of our method.

Dynamic obstacle avoidance for quadrotors with event cameras

Monocular Event-Based Vision for Obstacle Avoidance with a Quadrotor

Autonomous Flights in Dynamic Environments with Onboard Vision

Robust Vision-based Obstacle Avoidance for Micro Aerial Vehicles in Dynamic Environments

Fusing Stereopsis & Corner Sparse Optical Flow for Real-time Obstacle Avoidance of Unmanned Aerial Vehicle

Avoiding dynamic small obstacles with onboard sensing and computating on aerial robots

A real-time dynamic obstacle tracking and mapping system for UAV navigation and collision avoidance with an RGB-D camera

Monocular Vision-Based Obstacle Detection/Avoidance For Unmanned Aerial Vehicles

Free as a Bird: Event-based Dynamic Sense-and-Avoid for Ornithopter Robot Flight

3-D Reciprocal Collision Avoidance on Physical Quadrotor Helicopters with On-Board Sensing for Relative Positioning

Obstacle Avoidance of Monocular Quadrotors with Depth Estimation

FAST-Dynamic-Vision: Detection and Tracking Dynamic Objects with Event and Depth Sensing

Obstacle Avoidance Using Stereo Camera

Risk-aware Trajectory Sampling for Quadrotor Obstacle Avoidance in Dynamic Environments

Taming Event Cameras with Bio-Inspired Architecture and Algorithm: A Case for Drone Obstacle Avoidance

Detecting and avoiding frontal obstacles from monocular camera for micro unmanned aerial vehicles

Real-time Quadrotor Navigation Through Planning in Depth Space in Unstructured Environments

EVReflex: Dense Time-to-Impact Prediction for Event-based Obstacle Avoidance

Monocular Vision-Based Dynamic Moving Obstacles Detection and Avoidance

Monocular Snapshot-based Sensing and Control of Hover, Takeoff, and Landing for a Low-cost Quadrotor

Monocular Depth Estimation for UAV Obstacle Avoidance