Abstract:Accurately and reliably perceiving the environment is a major challenge in autonomous driving and robotics research. Traditional vision-based methods often suffer from varying lighting conditions, occlusions, and complex environments. This paper addresses these challenges by combining a deep learning-based object detection algorithm, YOLOv8, with LiDAR data fusion technology. The principle of this combination is to merge the advantages of these technologies: YOLOv8 excels in real-time object detection and classification through RGB images, while LiDAR provides accurate distance measurement and 3D spatial information, regardless of lighting conditions. The integration aims to apply the high accuracy and robustness of YOLOv8 in identifying and classifying objects, as well as the depth data provided by LiDAR. This combination enhances the overall environmental perception, which is critical for the reliability and safety of autonomous systems. However, this fusion brings some research challenges, including data calibration between different sensors, filtering ground points from LiDAR point clouds, and managing the computational complexity of processing large datasets. This paper presents a comprehensive approach to address these challenges. Firstly, a simple algorithm is introduced to filter out ground points from LiDAR point clouds, which are essential for accurate object detection, by setting different threshold heights based on the terrain. Secondly, YOLOv8, trained on a customized dataset, is utilized for object detection in images, generating 2D bounding boxes around detected objects. Thirdly, a calibration algorithm is developed to transform 3D LiDAR coordinates to image pixel coordinates, which are vital for correlating LiDAR data with image-based object detection results. Fourthly, a method for clustering different objects based on the fused data is proposed, followed by an object tracking algorithm to compute the 3D poses of objects and their relative distances from a robot. The Agilex Scout Mini robot, equipped with Velodyne 16-channel LiDAR and an Intel D435 camera, is employed for data collection and experimentation. Finally, the experimental results validate the effectiveness of the proposed algorithms and methods.

High-Precision Indoor Robot Dynamic Obstacle Detection with Laser and Camera

Obstacle Detection and Map Reconstruction Using Multi-Layer Laser Radar

Real-time dynamic obstacle detection and tracking using 3D Lidar

Robust Mobile Robot Localization By Fusing Laser And Sonar

A Implementation Method of Indoor SLAM Using Velodyne Laser Radar for Mobile Robot

Map building for indoor environment with laser range scanner

An Object Localization System Using Monocular Camera and Two-Axis-Controlled Laser Ranging Sensor for Mobile Robot

Onboard dynamic-object detection and tracking for autonomous robot navigation with RGB-D camera

Indoor Robot SLAM with Multi-Sensor Fusion

A real-time dynamic obstacle tracking and mapping system for UAV navigation and collision avoidance with an RGB-D camera

Real-Time Avoidance Strategy of Dynamic Obstacles via Half Model-Free Detection and Tracking With 2D Lidar for Mobile Robots

Robot visual perception and autonomous obstacle avoidance based on deep learning

Real-time 3D mapping using a 2D laser scanner and IMU-aided visual SLAM

An Advanced Approach to Object Detection and Tracking in Robotics and Autonomous Vehicles Using YOLOv8 and LiDAR Data Fusion

An SLAM Algorithm Based on Laser Radar and Vision Fusion with Loop Detection Optimization

Monocular Camera-based Complex Obstacle Avoidance via Efficient Deep Reinforcement Learning

A Monocular Vision Obstacle Avoidance Method Applied to Indoor Tracking Robot

A Low-Cost and Robust Multi-Sensor Data Fusion Scheme for Heterogeneous Multi-Robot Cooperative Positioning in Indoor Environments

Laser-Based People Detection and Obstacle Avoidance for a Hospital Transport Robot

A Dynamic Object Removing 3D Reconstruction System Based on Multi-Sensor Fusion