Abstract:Accurately and reliably perceiving the environment is a major challenge in autonomous driving and robotics research. Traditional vision-based methods often suffer from varying lighting conditions, occlusions, and complex environments. This paper addresses these challenges by combining a deep learning-based object detection algorithm, YOLOv8, with LiDAR data fusion technology. The principle of this combination is to merge the advantages of these technologies: YOLOv8 excels in real-time object detection and classification through RGB images, while LiDAR provides accurate distance measurement and 3D spatial information, regardless of lighting conditions. The integration aims to apply the high accuracy and robustness of YOLOv8 in identifying and classifying objects, as well as the depth data provided by LiDAR. This combination enhances the overall environmental perception, which is critical for the reliability and safety of autonomous systems. However, this fusion brings some research challenges, including data calibration between different sensors, filtering ground points from LiDAR point clouds, and managing the computational complexity of processing large datasets. This paper presents a comprehensive approach to address these challenges. Firstly, a simple algorithm is introduced to filter out ground points from LiDAR point clouds, which are essential for accurate object detection, by setting different threshold heights based on the terrain. Secondly, YOLOv8, trained on a customized dataset, is utilized for object detection in images, generating 2D bounding boxes around detected objects. Thirdly, a calibration algorithm is developed to transform 3D LiDAR coordinates to image pixel coordinates, which are vital for correlating LiDAR data with image-based object detection results. Fourthly, a method for clustering different objects based on the fused data is proposed, followed by an object tracking algorithm to compute the 3D poses of objects and their relative distances from a robot. The Agilex Scout Mini robot, equipped with Velodyne 16-channel LiDAR and an Intel D435 camera, is employed for data collection and experimentation. Finally, the experimental results validate the effectiveness of the proposed algorithms and methods.

Research of a Foreign Objects Detection Fusion Algorithm Using Video and LIDAR in FAO System

Fusing LiDAR and Radar with Pillars Attention for 3D Object Detection

Obstacle Detection by ALV Using Two 2D Laser Range Finders

Fusion Algorithm of Laser-Point Clouds and Optical Images

Adaptive Switching Spatial-Temporal Fusion Detection for Remote Flying Drones

Automatic Obstacle Detection Method for the Train Based on Deep Learning

Foreign object detection in urban rail transit based on deep differentiation segmentation neural network

The Visibility of Multisensor Fusion Technology in Public Art Design

Research and Application of the Obstacle Avoidance System for High-Speed Railway Tunnel Lining Inspection Train Based on Integrated 3D LiDAR and 2D Camera Machine Vision Technology

Deep LiDAR-Radar-Visual Fusion for Object Detection in Urban Environments

Design And Implementation of Laser Radar-Based Railway Foreign Object Intrusion Detection System

LiDAR and Camera Detection Fusion in a Real Time Industrial Multi-Sensor Collision Avoidance System

Real time object detection using LiDAR and camera fusion for autonomous driving

Railway obstacle intrusion warning mechanism integrating YOLO-based detection and risk assessment

Spatial Attention Fusion for Obstacle Detection Using MmWave Radar and Vision Sensor

ROFusion: Efficient Object Detection using Hybrid Point-wise Radar-Optical Fusion

A Novel High-Precision Railway Obstacle Detection Algorithm Based on 3D LiDAR

An Advanced Approach to Object Detection and Tracking in Robotics and Autonomous Vehicles Using YOLOv8 and LiDAR Data Fusion

Design and Implementation of Intelligent Security Robot Based on Lidar and Vision Fusion*

A Novel Multi-Sensor Fusion Based Object Detection and Recognition Algorithm for Intelligent Assisted Driving

RI-Fusion: 3D Object Detection Using Enhanced Point Features With Range-Image Fusion for Autonomous Driving.