Abstract:We address the problem of multi-modal object tracking in video and explore various options available for fusing the complementary information conveyed by the visible (RGB) and thermal infrared (TIR) modalities, including pixel-level, feature-level and decision-level fusion. Specifically, in contrast to the existing approaches, we propose and develop the paradigm for combining multi-modal information for image fusion at pixel level. At the feature level, two different kinds of fusion strategies are investigated for completeness, i.e. , the attention-based online fusion strategy and the offline-trained fusion block. At the decision level, a novel fusion strategy is put forward, inspired by the success of the simple averaging configuration which has shown so much promise. The effectiveness of the proposed decision-level fusion strategy owes to a number of innovative contributions, including a dynamic weighting of the RGB and TIR contributions and a linear template update operation. A variant of the proposed decision fusion method produced the winning tracker at the Visual Object Tracking Challenge 2020 (VOT-RGBT2020). A comprehensive comparison of the innovative pixel and feature-level fusion strategies with the proposed decision-level fusion method highlights the advantages fusing multimodal information at the decision score level. Extensive experimental results on five challenging datasets, i.e. , GTOT, VOT-RGBT2019, RGBT234, LasHeR and VOT-RGBT2020, demonstrate the effectiveness and robustness of the proposed method, compared to the state-of-the-art approaches. The Code is available at https://github.com/Zhangyong-Tang/DFAT .

Fusion Tracking in Color and Infrared Images Using Sequential Belief Propagation

Fusion Tracking in Color and Infrared Images Using Joint Sparse Representation

Infrared and Visible Image Fusion Via Variational Bayesian Approximation Method

Tracking an Object over 200 FPS with the Fusion of Prior Probability and Kalman Filter

Visual tracking using the kernel based particle filter and color distribution

Fusion of infrared and visual images through multiscale hybrid unidirectional total variation

Real Time Object Tracking Using Fused Color And Edge Cues

Vehicle Tracking Using Stochastic Fusion-Based Particle Filter

The Effect of Infrared and Visible Image Fusion on Object Tracking Using Correlation Matching

Real-Time Speaker Tracking Using Particle Filter Sensor Fusion

Visual Tracking Via Adaptive Interactive Fusion

Object fusion tracking for RGB-T images via channel swapping and modal mutual attention

Efficient Visual Tracking Using Particle Filter with Incremental Likelihood Calculation

Infrared Object Tracking Algorithm with Temporal Features Fusion Siamese Network

A robust infrared and visible image fusion framework via multi-receptive-field attention and color visual perception

Exploring fusion strategies for accurate RGBT visual object tracking

Anti-occlusion Particle Filter Object-Tracking Method Based on Feature Fusion

DSiamMFT: An RGB-T fusion tracking method via dynamic Siamese networks using multi-layer feature fusion

Bayesian fusion for infrared and visible images

Infrared-visible Image Fusion Using Accelerated Convergent Convolutional Dictionary Learning

Infrared and Visible Image Fusion Based on Filtering Enhancement