Abstract:3D Multi-object tracking (MOT) ensures consistency during continuous dynamic detection, conducive to subsequent motion planning and navigation tasks in autonomous driving. However, camera-based methods suffer in the case of occlusions and it can be challenging to track the irregular motion of objects for LiDAR-based methods accurately. Some fusion methods work well but do not consider the untrustworthy issue of appearance features under occlusion. At the same time, the false detection problem also significantly affects tracking. As such, we propose a novel camera-LiDAR fusion 3D MOT framework based on Combined Appearance-Motion Optimization (CAMO-MOT), which uses both camera and LiDAR data and significantly reduces tracking failures caused by occlusion and false detection. For occlusion problems, we are the first to propose an occlusion head to select the best object appearance features multiple times effectively, reducing the influence of occlusions. To decrease the impact of false detection in tracking, we design a motion cost matrix based on confidence scores which improve the positioning and object prediction accuracy in 3D space. As existing multi-object tracking methods always evaluate each category separately and do not consider the mismatch between objects of different categories, we also propose to build a multi-category cost to implement multi-object tracking in multi-category scenes. A series of validation experiments are conducted on the KITTI and nuScenes tracking benchmarks. Our proposed method achieves state-of-the-art performance with 79.99% HOTA and the lowest identity switches (IDS) value (23 for Car and 137 for Pedestrian) among all multi-modal MOT methods on the KITTI test dataset. And our method achieves state-of-the-art performance among all algorithms on the nuScenes test dataset with 75.3% AMOTA.

Multi-3D-Object Tracking by Fusing RGB and 3D-LiDAR Data

Online Multi-Object Tracking from A Bird's-Eye View by Fusion of Millimeter-Wave Radar and Vision

A Tracking-By-Detection Based 3D Multiple Object Tracking for Autonomous Driving

Multi-Object Tracking with Camera-LiDAR Fusion for Autonomous Driving

Tightly Coupled Integration of LiDAR and Vision for 3D Multiobject Tracking

3D Multiple Object Tracking with Multi-modal Fusion of Low-cost Sensors for Autonomous Driving.

Cross-Modality 3D Multi-Object Tracking under Adverse Weather Via Adaptive Hard Sample Mining

Joint Multi-Object Detection and Tracking with Camera-LiDAR Fusion for Autonomous Driving

Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Driving

Multimodal Multiobject Tracking by Fusing Deep Appearance Features and Motion Information

Object-Level Pseudo-3D Lifting for Distance-Aware Tracking

3D Multiple Extended Object Tracking by Fusing Roadside Radar and Camera Sensors

DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion With Deep Association

CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking With Camera-LiDAR Fusion

Multiple-object Tracking Based on Monocular Camera and 3-D Lidar Fusion for Autonomous Vehicles

Tracking Objects with 3D Representation from Videos

PF-MOT: Probability Fusion Based 3D Multi-Object Tracking for Autonomous Vehicles

Boost Correlation Features with 3D-MiIoU-Based Camera-LiDAR Fusion for MODT in Autonomous Driving

Interactive Multi-Scale Fusion of 2D and 3D Features for Multi-Object Vehicle Tracking

3D Multi-Object Tracking Based on Radar-Camera Fusion

StrongFusionMOT: A Multi-Object Tracking Method Based on LiDAR-Camera Fusion