Abstract:Multiple object tracking (MOT) is a significant task in achieving autonomous driving. Traditional works attempt to complete this task, either based on point clouds (PC) collected by LiDAR, or based on images captured from cameras. However, relying on one single sensor is not robust enough, because it might fail during the tracking process. On the other hand, feature fusion from multiple modalities contributes to the improvement of accuracy. As a result, new techniques based on different sensors integrating features from multiple modalities are being developed. Texture information from RGB cameras and 3D structure information from Lidar have respective advantages under different circumstances. However, it's not easy to achieve effective feature fusion because of completely distinct information modalities. Previous fusion methods usually fuse the top-level features after the backbones extract the features from different modalities. In this paper, we first introduce PointNet++ to obtain multi-scale deep representations of point cloud to make it adaptive to our proposed Interactive Feature Fusion between multi-scale features of images and point clouds. Specifically, through multi-scale interactive query and fusion between pixel-level and point-level features, our method, can obtain more distinguishing features to improve the performance of multiple object tracking. Besides, we explore the effectiveness of pre-training on each single modality and fine-tuning on the fusion-based model. The experimental results demonstrate that our method can achieve good performance on the KITTI benchmark and outperform other approaches without using multi-scale feature fusion. Moreover, the ablation studies indicates the effectiveness of multi-scale feature fusion and pre-training on single modality.

Multi-Stage Fusion for Event-based Multimodal Tracker

Intensity/Inertial Integration-Aided Feature Tracking on Event Cameras

Frame-Event Alignment and Fusion Network for High Frame Rate Tracking

TENet: Targetness Entanglement Incorporating with Multi-Scale Pooling and Mutually-Guided Fusion for RGB-E Object Tracking

Cross Fusion RGB-T Tracking with Bi-directional Adapter

Unsupervised RGB-T object tracking with attentional multi-modal feature fusion

Object fusion tracking for RGB-T images via channel swapping and modal mutual attention

Learning Dual-Fused Modality-Aware Representations for RGBD Tracking

Standard and Event Cameras Fusion for Feature Tracking

Exploring fusion strategies for accurate RGBT visual object tracking

MMF-Track: Multi-modal Multi-level Fusion for 3D Single Object Tracking

QueryTrack: Joint-Modality Query Fusion Network for RGBT Tracking

Mamba-FETrack: Frame-Event Tracking via State Space Model

SiamEFT: adaptive-time feature extraction hybrid network for RGBE multi-domain object tracking

Dynamic Subframe Splitting and Spatio-Temporal Motion Entangled Sparse Attention for RGB-E Tracking

Interactive Multi-scale Fusion of 2D and 3D Features for Multi-object Tracking

Multi-Scale Feature Interactive Fusion Network for RGBT Tracking

Learning Modality Complementary Features with Mixed Attention Mechanism for RGB-T Tracking

Tracking Any Point with Frame-Event Fusion Network at High Frame Rate

Multi-domain collaborative feature representation for robust visual object tracking

Jointly Modeling Motion and Appearance Cues for Robust RGB-T Tracking