Abstract:LiDAR has become one of the primary sensors in robotics and autonomous system for high-accuracy situational awareness. In recent years, multi-modal LiDAR systems emerged, and among them, LiDAR-as-a-camera sensors provide not only 3D point clouds but also fixed-resolution 360°panoramic images by encoding either depth, reflectivity, or near-infrared light in the image pixels. This potentially brings computer vision capabilities on top of the potential of LiDAR itself. In this paper, we are specifically interested in utilizing LiDARs and LiDAR-generated images for tracking Unmanned Aerial Vehicles (UAVs) in real-time which can benefit applications including docking, remote identification, or counter-UAV systems, among others. This is, to the best of our knowledge, the first work that explores the possibility of fusing the images and point cloud generated by a single LiDAR sensor to track a UAV without a priori known initialized position. We trained a custom YOLOv5 model for detecting UAVs based on the panoramic images collected in an indoor experiment arena with a MOCAP system. By integrating with the point cloud, we are able to continuously provide the position of the UAV. Our experiment demonstrated the effectiveness of the proposed UAV tracking approach compared with methods based only on point clouds or images. Additionally, we evaluated the real-time performance of our approach on the Nvidia Jetson Nano, a popular mobile computing platform.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to use the images and point cloud data generated by LiDAR to track unmanned aerial vehicles (UAVs) in real - time in an environment where GNSS signals are unavailable or interfered. Specifically, the paper explores how to fuse the images and point cloud generated by a single LiDAR sensor to achieve UAV tracking without prior knowledge of the initial position of the UAV. This is the first research attempt to use this fusion method for UAV tracking. ### Background and Motivation In mobile robots and autonomous systems, high - precision situational awareness is a key task, which can support effective decision - making, navigation and control. In recent years, multi - modal LiDAR systems have gradually emerged. Among them, the "LiDAR - as - a - camera" sensor not only provides 3D point clouds, but also generates 360 - degree panoramic images with a fixed resolution. These images can encode depth, reflectivity or near - infrared light. This enables computer vision capabilities to be further enhanced on the basis of LiDAR. ### Research Objectives The main objective of this paper is to use the images and point cloud data generated by LiDAR to achieve real - time UAV tracking. Specifically, the researchers used a customized YOLOv5 model to detect UAVs in panoramic images collected based on an indoor experimental site, and continuously provided the position information of UAVs by integrating point cloud data. ### Method Overview 1. **Initializing the UAV Position**: - Use the YOLOv5 model to detect the position of the UAV in the signal image, thereby obtaining the region of interest (ROI). - Extract the corresponding ROI from the point cloud data, and extract the point cloud of the UAV through ground removal and point cloud clustering techniques, and estimate its initial position. 2. **Fusing Signal Images and Point Cloud Data**: - Perform object detection on each signal image to obtain a more accurate ROI. - If the YOLOv5 fails to detect the UAV, use the ROI predicted by the Kalman filter to separate the point cloud of the UAV. ### Experimental Results - **Trajectory Verification**: The effectiveness of the proposed method was verified by comparing it with methods using only point clouds or only signal images. The experimental results show that the fusion method performs best in terms of average absolute position error (APE) and velocity error. - **Real - Time Performance**: The real - time performance of the method was evaluated on the Nvidia Jetson Nano platform, and the results show that the method can operate effectively on the embedded platform. ### Conclusions This paper proposes a UAV tracking method based on the images and point cloud data generated by LiDAR. Different from the traditional LiDAR and camera fusion methods, this method does not require the calibration and pre - processing of external cameras, and has stronger adaptability to harsh environments. The experimental results show that this method has high accuracy and real - time performance in various application scenarios. ### Future Work Future work will include fusing the depth, signal and environmental images generated by Ouster LiDAR, as well as traditional RGB images, and applying them in fields such as simultaneous localization and mapping (SLAM), object detection and tracking.

UAV Tracking with Lidar as a Camera Sensors in GNSS-Denied Environments

UAV Tracking with Solid-State Lidars:Dynamic Multi-Frequency Scan Integration

Real-Time Multi-Modal Active Vision for Object Detection on UAVs Equipped With Limited Field of View LiDAR and Camera

A Vision-based UAV Tracker Aiming at Aerial Targets

Adaptive Lidar Scan Frame Integration: Tracking Known MAVs in 3D Point Clouds

Towards Robust UAV Tracking in GNSS-Denied Environments: A Multi-LiDAR Multi-UAV Dataset

LCDL: Towards Dynamic Localization for Autonomous Landing of Unmanned Aerial Vehicle Based on LiDAR-Camera Fusion

Robust Visual-Lidar Simultaneous Localization and Mapping System for UAV

A Real-Time Tracking Algorithm for Multi-Target UAV Based on Deep Learning.

Deep Learning based Multi-Modal Sensing for Tracking and State Extraction of Small Quadcopters

Evaluating the Performance of Multi-Scan Integration for UAV LiDAR-based Tracking

Real‐time visual tracking design for an unmanned aerial vehicle in cluttered environments

A Real-time Positioning Model for UAV's Patrolling Images Based on Airborne LiDAR Point Cloud Fusion

Real-Time Visual Tracking of Moving Targets Using a Low-Cost Unmanned Aerial Vehicle with a 3-Axis Stabilized Gimbal System

Camera-Based Local and Global Target Detection, Tracking, and Localization Techniques for UAVs

A Deep Learning Approach of Intrusion Detection and Tracking with UAV-Based 360° Camera and 3-Axis Gimbal

Unsupervised UAV 3D Trajectories Estimation with Sparse Point Clouds

Real-time Active Detection of Targets and Path Planning Using UAVs

Person Tracking and Frontal Face Capture with UAV

UAV Based Indoor Localization and Objection Detection

LiDAR Depth Cluster Active Detection and Localization for a UAV with Partial Information Loss in GNSS