Abstract:In this paper, we present a novel 2D⁻3D pedestrian tracker designed for applications in autonomous vehicles. The system operates on a tracking by detection principle and can track multiple pedestrians in complex urban traffic situations. By using a behavioral motion model and a non-parametric distribution as state model, we are able to accurately track unpredictable pedestrian motion in the presence of heavy occlusion. Tracking is performed independently, on the image and ground plane, in global, motion compensated coordinates. We employ Camera and LiDAR data fusion to solve the association problem where the optimal solution is found by matching 2D and 3D detections to tracks using a joint log-likelihood observation model. Each 2D⁻3D particle filter then updates their state from associated observations and a behavioral motion model. Each particle moves independently following the pedestrian motion parameters which we learned offline from an annotated training dataset. Temporal stability of the state variables is achieved by modeling each track as a Markov Decision Process with probabilistic state transition properties. A novel track management system then handles high level actions such as track creation, deletion and interaction. Using a probabilistic track score the track manager can cull false and ambiguous detections while updating tracks with detections from actual pedestrians. Our system is implemented on a GPU and exploits the massively parallelizable nature of particle filters. Due to the Markovian nature of our track representation, the system achieves real-time performance operating with a minimal memory footprint. Exhaustive and independent evaluation of our tracker was performed by the KITTI benchmark server, where it was tested against a wide variety of unknown pedestrian tracking situations. On this realistic benchmark, we outperform all published pedestrian trackers in a multitude of tracking metrics.

A Pedestrian Detection and Tracking Framework for Autonomous Cars: Efficient Fusion of Camera and LiDAR Data

Online Multipedestrian Tracking Based on Fused Detections of Millimeter Wave Radar and Vision

Fully convolutional neural networks for LIDAR–camera fusion for pedestrian detection in autonomous vehicle

Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar

Pedestrian Tracking Based on Laser and Image Data Fusion.

Boosting Cnn-Based Pedestrian Detection Via 3d Lidar Fusion In Autonomous Driving

A Pedestrian Detection Algorithm Based on Score Fusion for Multi-LiDAR Systems

Deep Learning-Based Pedestrian Detection Using RGB Images and Sparse LiDAR Point Clouds

Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving

An Integrated Deep Learning Framework for Occluded Pedestrian Tracking.

Real-Time Vehicle Detection Framework Based on the Fusion of LiDAR and Camera

Pedestrian Facial Attention Detection Using Deep Fusion and Multi-modal Fusion Classifier

Pedestrian Detection by Fusion of RGB and Infrared Images in Low-Light Environment

Real time object detection using LiDAR and camera fusion for autonomous driving

Three-Dimensional Pedestrian Detection by Fusing Image Semantics and Point Cloud Spatial Visibility Features

Pedestrian Detection Aided by Fusion of Binocular Information

Spatio-contextual deep network-based multimodal pedestrian detection for autonomous driving

A Fast RetinaNet Fusion Framework for Multi-Spectral Pedestrian Detection

An Efficient 3D Pedestrian Detector with Calibrated RGB Camera and 3D LiDAR.

Behavioral Pedestrian Tracking Using a Camera and LiDAR Sensors on a Moving Vehicle