Abstract:Existing datasets for RGB-DVS tracking are collected with DVS346 camera and their resolution ($346 \times 260$) is low for practical applications. Actually, only visible cameras are deployed in many practical systems, and the newly designed neuromorphic cameras may have different resolutions. The latest neuromorphic sensors can output high-definition event streams, but it is very difficult to achieve strict alignment between events and frames on both spatial and temporal views. Therefore, how to achieve accurate tracking with unaligned neuromorphic and visible sensors is a valuable but unresearched problem. In this work, we formally propose the task of object tracking using unaligned neuromorphic and visible cameras. We build the first unaligned frame-event dataset CRSOT collected with a specially built data acquisition system, which contains 1,030 high-definition RGB-Event video pairs, 304,974 video frames. In addition, we propose a novel unaligned object tracking framework that can realize robust tracking even using the loosely aligned RGB-Event data. Specifically, we extract the template and search regions of RGB and Event data and feed them into a unified ViT backbone for feature embedding. Then, we propose uncertainty perception modules to encode the RGB and Event features, respectively, then, we propose a modality uncertainty fusion module to aggregate the two modalities. These three branches are jointly optimized in the training phase. Extensive experiments demonstrate that our tracker can collaborate the dual modalities for high-performance tracking even without strictly temporal and spatial alignment. The source code, dataset, and pre-trained models will be released at

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve high - precision object tracking in the case of misalignment between neuromorphic cameras and visible cameras with different resolutions. Specifically, most of the existing RGB - DVS (Dynamic Visual Sensor) tracking datasets are collected using low - resolution DVS346 cameras, which limits their performance in practical applications. In addition, since newly - designed neuromorphic cameras can output high - definition event streams but may have different resolutions, it becomes very difficult to strictly align event streams and RGB frames in space and time. Therefore, how to use these sensors with different resolutions to achieve efficient object tracking without strict spatio - temporal alignment is a valuable but not yet fully studied problem. To solve this problem, the paper makes the following contributions: 1. **Propose a new task setting**: Introduce a new task setting for object tracking using misaligned neuromorphic and visible cameras, providing new ideas for introducing neuromorphic cameras into visible - camera - based surveillance systems. 2. **Construct a high - resolution dataset**: Create the first high - resolution, large - scale, high - quality dataset CRSOT for cross - resolution single - object tracking, which contains 1,030 video sequences and 304,974 RGB frames, and divide it into a training set and a test set. 3. **Propose a new tracking framework**: Design a novel misaligned object tracking framework that can achieve robust tracking performance even when using misaligned RGB - Event data. This framework effectively deals with the challenges brought by misaligned data by introducing the Modal Uncertainty Perception Module (MDUP), the Cross - Modal Uncertainty Perception Module (CMDUP), and the Modal Uncertainty Fusion Module (MUF). The paper verifies the effectiveness of the proposed framework on multiple benchmark datasets through extensive experiments, hoping to attract more researchers to pay attention to the misaligned dual - modal tracking problem.

CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Cameras

Exploit Spatiotemporal Contextual Information for 3D Single Object Tracking Via Memory Networks

Unidirectional Cross-Modal Fusion for RGB-T Tracking

Cross-Modal Object Tracking: Modality-Aware Representations and a Unified Benchmark

Adaptive Resolution Optimization and Tracklet Reliability Assessment for Efficient Multi-Object Tracking

Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline

VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows

SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking

Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric

AMATrack: A Unified Network With Asymmetric Multimodal Mixed Attention for RGBD Tracking

CIRNet: an Improved RGBT Tracking Via Cross-Modality Interaction and Re-Identification

Cross-Modal Semi-Dense 6-DoF Tracking of an Event Camera in Challenging Conditions

Robust Visual Object Tracking with Two-Stream Residual Convolutional Networks

Single-Model and Any-Modality for Video Object Tracking

Unsupervised RGB-T object tracking with attentional multi-modal feature fusion

Residual Learning-Based Two-Stream Network for RGB-T Object Tracking

OmniPose6D: Towards Short-Term Object Pose Tracking in Dynamic Scenes from Monocular RGB

XTrack: Multimodal Training Boosts RGB-X Video Object Trackers

Learning Modality Residual Regularized Correlation Filter for Real Time RGB-T Tracking

Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers