Abstract:Multimodal visual object tracking (VOT) starts to replace a single modality by providing more complementary details of the target. In RGB plus depth (RGBD) tracking tasks, prevailing trackers extract RGB and depth features with separate feature extraction backbones in parallel structures and fuse the features for target prediction, where RGB is treated as the dominant modality, while the depth information are the assistant one. Their oversight lies in neglecting the distinct information-carrying capacities of each modality, and the fact that RGB and depth features contribute dynamically and variably to the tracking process. In response to these challenges, we have introduced a unified RGBD tracking network called AMATrack, which is designed to simultaneously extract features and investigate interactions between the individual RGB and depth images. Initially, the depth image is integrated with the RGB image to form the fused template and search region separately, ensuring maximal preservation of information from both modalities. Subsequently, we introduce an asymmetric mixed attention (AMA) module within the encoder layers. This module incorporates self-attention for maintaining intramodality features and cross-attention for minimizing discrepancies in intermodality features. Simultaneously, it removes correlations with nontarget patches to prevent adverse effects from background areas. To address the additional computational load during training and inference, we have customized a target token pruning (TTP) technique dubbed TTP. This strategy filters out patch tokens least relevant to the target before generating the final search region, thereby enhancing computational efficiency. In conclusion, our proposed AMATrack attains an F-score of 61.8% and 75.8% on the DepthTrack dataset and CDTB dataset, respectively, while operating at a speed of 73.01 frames/s (FPS). This performance surpasses that of the leading RGBD trackers in meeting real-time demands. These outcomes demonstrate the efficacy and superiority of our proposed network.

ARKitTrack: A New Diverse Dataset for Tracking Using Mobile RGB-D Data

ARKitScenes - A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data

Digital Twin Tracking Dataset (DTTD): A New RGB+Depth 3D Dataset for Longer-Range Object Tracking Applications

DepthTrack : Unveiling the Power of RGBD Tracking

Tracking Revisited using RGBD Camera: Baseline and Benchmark

RGBD Object Tracking: An In-depth Review

RGB-T Object Tracking:Benchmark and Baseline

Robust Keyframe-based Dense SLAM with an RGB-D Camera.

RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videos

RGB-T Object Tracking: Benchmark and Baseline

Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline

Robust and Accurate RGB-D Reconstruction With Line Feature Constraints

RGB-D-E: Event Camera Calibration for Fast 6-DOF object Tracking

Caltech Aerial RGB-Thermal Dataset in the Wild

OmniPose6D: Towards Short-Term Object Pose Tracking in Dynamic Scenes from Monocular RGB

Autoregressive Visual Tracking

Visual Object Tracking on Multi-modal RGB-D Videos: A Review

Robust 6DoF Pose Estimation Against Depth Noise and a Comprehensive Evaluation on a Mobile Dataset

MMPTRACK: Large-scale Densely Annotated Multi-camera Multiple People Tracking Benchmark

RAV4D: A Radar-Audia-Visual Dataset for Indoor Multi - Person Tracking

AMATrack: A Unified Network With Asymmetric Multimodal Mixed Attention for RGBD Tracking