Abstract:Multimodal sensing has proven valuable for visual tracking, as different sensor types offer unique strengths in handling one specific challenging scene where object appearance varies. While a generalist model capable of leveraging all modalities would be ideal, development is hindered by data sparsity, typically in practice, only one modality is available at a time. Therefore, it is crucial to ensure and achieve that knowledge gained from multimodal sensing -- such as identifying relevant features and regions -- is effectively shared, even when certain modalities are unavailable at inference. We venture with a simple assumption: similar samples across different modalities have more knowledge to share than otherwise. To implement this, we employ a ``weak" classifier tasked with distinguishing between modalities. More specifically, if the classifier ``fails" to accurately identify the modality of the given sample, this signals an opportunity for cross-modal knowledge sharing. Intuitively, knowledge transfer is facilitated whenever a sample from one modality is sufficiently close and aligned with another. Technically, we achieve this by routing samples from one modality to the expert of the others, within a mixture-of-experts framework designed for multimodal video object tracking. During the inference, the expert of the respective modality is chosen, which we show to benefit from the multimodal knowledge available during training, thanks to the proposed method. Through the exhaustive experiments that use only paired RGB-E, RGB-D, and RGB-T during training, we showcase the benefit of the proposed method for RGB-X tracker during inference, with an average +3\% precision improvement over the current SOTA. Our source code is publicly available at <a class="link-external link-https" href="https://github.com/supertyd/XTrack/tree/main" rel="external noopener nofollow">this https URL</a>.

How Does Colour Predict Multiple Object Tracking Performance? the Role of Surface Feature in Attentive Tracking

How Do Humans Perform in Multiple Object Tracking with Unstable Features.

APPTracker: Improving Tracking Multiple Objects in Low-Frame-Rate Videos

Exploit Spatiotemporal Contextual Information for 3D Single Object Tracking Via Memory Networks

The Role of Spatial Configuration in Multiple Identity Tracking

Tracking Every Thing in the Wild.

Object-Level Pseudo-3D Lifting for Distance-Aware Tracking

Distractor-aware discrimination learning for online multiple object tracking

Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking

MotionTrack: Learning Motion Predictor for Multiple Object Tracking

Dense Scene Multiple Object Tracking with Box-Plane Matching

Multiple object tracking with appearance feature prediction and similarity fusion

All eyes on relevance: strategic allocation of attention as a result of feature-based task demands in multiple object tracking

XTrack: Multimodal Training Boosts RGB-X Video Object Trackers

Active-matting-based Object Tracking with Color Cues

Object tracking by color distribution fields with adaptive hierarchical structure

Feature enhancement and coarse-to-fine detection for RGB-D tracking

Spatial-Semantic and Temporal Attention Mechanism-Based Online Multi-Object Tracking

Neural correlates of multiple object tracking strategies

Multi-Object Tracking Algorithm for RGB-D Images Based on Asymmetric Dual Siamese Networks

A Review of Detection-Related Multiple Object Tracking in Recent Times