Abstract:Multimodal vision-language (VL) learning has noticeably pushed the tendency toward generic intelligence owing to emerging large foundation models. However, tracking, as a fundamental vision problem, surprisingly enjoys less bonus from recent flourishing VL learning. We argue that the reasons are two-fold: the lack of large-scale vision-language annotated videos and ineffective vision-language interaction learning of current works. These nuisances motivate us to design more effective vision-language representation for tracking, meanwhile constructing a large database with language annotation for model learning. Particularly, in this paper, we first propose a general attribute annotation strategy to decorate videos in six popular tracking benchmarks, which contributes a large-scale vision-language tracking database with more than 23,000 videos. We then introduce a novel framework to improve tracking by learning a unified-adaptive VL representation, where the cores are the proposed asymmetric architecture search and modality mixer (ModaMixer). To further improve VL representation, we introduce a contrastive loss to align different modalities. To thoroughly evidence the effectiveness of our method, we integrate the proposed framework on three tracking methods with different designs, i.e., the CNN-based SiamCAR, the Transformer-based OSTrack, and the hybrid structure TransT. The experiments demonstrate that our framework can significantly improve all baselines on six benchmarks. Besides empirical results, we theoretically analyze our approach to show its rationality. By revealing the potential of VL representation, we expect the community to divert more attention to VL tracking and hope to open more possibilities for future tracking with diversified multimodal messages.

Cross-modal Target Retrieval for Tracking by Natural Language

Context-Aware Integration of Language and Visual References for Natural Language Tracking

Language-guided Dual-modal Local Correspondence for Single Object Tracking

TLPG-Tracker: Joint Learning of Target Localization and Proposal Generation for Visual Tracking.

Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark

Multi-features Guided Robust Visual Tracking.

Real-time Visual Object Tracking with Natural Language Description

Beyond Visual Cues: Synchronously Exploring Target-Centric Semantics for Vision-Language Tracking

Unifying Visual and Vision-Language Tracking via Contrastive Learning

Multi-modal Understanding and Generation for Object Tracking

Unified Transformer with Isomorphic Branches for Natural Language Tracking

TDCL: Dense Semantic Contrastive Learning for Vision-Language Tracking

Cross-Modal Object Tracking: Modality-Aware Representations and a Unified Benchmark

Describe and Attend to Track: Learning Natural Language guided Structural Representation and Visual Attention for Object Tracking

Tracking by Joint Local and Global Search: A Target-Aware Attention-Based Approach

Generalizing Multiple Object Tracking to Unseen Domains by Introducing Natural Language Representation

Tell Me What to Track: Infusing Robust Language Guidance for Enhanced Referring Multi-Object Tracking

Transformer Vision-Language Tracking via Proxy Token Guided Cross-Modal Fusion

Multi-Modal Visual Tracking Based on Textual Generation

Effective Local and Global Search for Fast Long-term Tracking

Divert More Attention to Vision-Language Object Tracking