Abstract:Most existing multi-object tracking methods typically learn visual tracking features via maximizing dis-similarities of different instances and minimizing similarities of the same instance. While such a feature learning scheme achieves promising performance, learning discriminative features solely based on visual information is challenging especially in case of environmental interference such as occlusion, blur and domain variance. In this work, we argue that multi-modal language-driven features provide complementary information to classical visual features, thereby aiding in improving the robustness to such environmental interference. To this end, we propose a new multi-object tracking framework, named LG-MOT, that explicitly leverages language information at different levels of granularity (scene-and instance-level) and combines it with standard visual features to obtain discriminative representations. To develop LG-MOT, we annotate existing MOT datasets with scene-and instance-level language descriptions. We then encode both instance-and scene-level language information into high-dimensional embeddings, which are utilized to guide the visual features during training. At inference, our LG-MOT uses the standard visual features without relying on annotated language descriptions. Extensive experiments on three benchmarks, MOT17, DanceTrack and SportsMOT, reveal the merits of the proposed contributions leading to state-of-the-art performance. On the DanceTrack test set, our LG-MOT achieves an absolute gain of 2.2\% in terms of target object association (IDF1 score), compared to the baseline using only visual features. Further, our LG-MOT exhibits strong cross-domain generalizability. The dataset and code will be available at ~\url{<a class="link-external link-https" href="https://github.com/WesLee88524/LG-MOT" rel="external noopener nofollow">this https URL</a>}.

VGT-MOT: Visibility-Guided Tracking for Online Multiple-Object Tracking

Exploit the Connectivity: Multi-Object Tracking with TrackletNet

Exploit the Connectivity

Online Multi-Object Tracking from A Bird's-Eye View by Fusion of Millimeter-Wave Radar and Vision

MAT: Motion-Aware Multi-Object Tracking

Online Multi-Object Tracking Using CNN-based Single Object Tracker with Spatial-Temporal Attention Mechanism

MotionTrack: Learning Robust Short-term and Long-term Motions for Multi-Object Tracking

Towards Real-Time Multi-Object Tracking

Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking

Spatial-Semantic and Temporal Attention Mechanism-Based Online Multi-Object Tracking

Multi-Granularity Language-Guided Multi-Object Tracking

Occlusion Handling Based on Motion Estimation for Multi-Object Tracking

Multi-object tracking algorithm based on interactive attention network and adaptive trajectory reconnection

Visible-Thermal Multiple Object Tracking: Large-scale Video Dataset and Progressive Fusion Approach

TPTrack: Strengthening tracking-by-detection methods from tracklet processing perspectives

VariabilityTrack:Multi-Object Tracking with Variable Speed Object Movement

CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking With Camera-LiDAR Fusion

Track Initialization and Re-Identification for~3D Multi-View Multi-Object Tracking

DyGLIP: A Dynamic Graph Model with Link Prediction for Accurate Multi-Camera Multiple Object Tracking

Online multiple object tracking based on fusing global and partial features

LaMOT: Language-Guided Multi-Object Tracking