Abstract:Domain adaptation (DA) has demonstrated significant promise for real-time nighttime unmanned aerial vehicle (UAV) tracking. However, the state-of-the-art (SOTA) DA still lacks the potential object with accurate pixel-level location and boundary to generate the high-quality target domain training sample. This key issue constrains the transfer learning of the real-time daytime SOTA trackers for challenging nighttime UAV tracking. Recently, the notable Segment Anything Model (SAM) has achieved a remarkable zero-shot generalization ability to discover abundant potential objects due to its huge data-driven training approach. To solve the aforementioned issue, this work proposes a novel SAM-powered DA framework for real-time nighttime UAV tracking, i.e., SAM-DA. Specifically, an innovative SAM-powered target domain training sample swelling is designed to determine enormous high-quality target domain training samples from every single raw nighttime image. This novel one-to-many generation significantly expands the high-quality target domain training sample for DA. Comprehensive experiments on extensive nighttime UAV videos prove the robustness and domain adaptability of SAM-DA for nighttime UAV tracking. Especially, compared to the SOTA DA, SAM-DA can achieve better performance with fewer raw nighttime images, i.e., the fewer-better training. This economized training approach facilitates the quick validation and deployment of algorithms for UAVs. The code is available at <a class="link-external link-https" href="https://github.com/vision4robotics/SAM-DA" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the domain adaptation problem in nighttime UAV (Unmanned Aerial Vehicle) target tracking. Specifically, existing domain adaptation methods struggle with nighttime images due to the lack of high-quality target domain training samples, making it difficult to effectively apply advanced daytime trackers to nighttime UAV tracking. Nighttime images are characterized by insufficient lighting, low contrast, and high noise, which make it challenging to extract precise pixel-level position and boundary information from these images. Therefore, generating a large number of high-quality target domain training samples to achieve robust day-to-night domain adaptation is an urgent problem to be solved. ### Solution To address the above issues, the authors propose a domain adaptation framework based on the Segment Anything Model (SAM) (SAM-DA). This framework leverages SAM's powerful zero-shot generalization capability to generate a large number of high-quality target domain training samples from each original nighttime image. The specific steps are as follows: 1. **SAM-Powered Target Domain Training Sample Expansion**: - Use SAM to automatically identify a large number of potential objects from each nighttime image and provide their precise pixel-level positions and boundaries. - Generate multiple target domain training samples through operations such as cropping and resizing, thus expanding a single nighttime image into multiple training samples. 2. **Tracking-Oriented Day-to-Night Domain Adaptation**: - Utilize the generated large number of high-quality target domain training samples to conduct tracking-oriented day-to-night domain adaptation training. - Improve the tracker's performance under nighttime conditions through feature alignment. ### Experimental Results Experimental results show that SAM-DA performs excellently in nighttime UAV tracking tasks, especially when using fewer original nighttime images, achieving better performance than existing methods. Specifically, it excels in the following aspects: - **Success Rate, Precision, and Normalized Precision**: On the DarkTrack2021 and NUT-L benchmark datasets, SAM-DA-Track significantly outperforms other methods in terms of success rate, precision, and normalized precision. - **Lighting Challenges**: Under extreme low-light conditions and lighting variations, SAM-DA-Track demonstrates stronger robustness. ### Main Contributions 1. **Proposed a novel domain adaptation framework (SAM-DA) based on SAM**, applying SAM to nighttime UAV tracking domain adaptation for the first time. 2. **Designed an innovative SAM-Powered target domain training sample expansion method**, capable of generating a large number of high-quality target domain training samples from each nighttime image. 3. **Validated the effectiveness and domain adaptation capability of SAM-DA through extensive experiments**, achieving better tracking performance even with fewer original nighttime images. In summary, this paper introduces SAM to solve the problem of generating high-quality target domain training samples in nighttime UAV tracking, significantly improving the tracker's performance under nighttime conditions.

SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation

Prompt-Driven Temporal Domain Adaptation for Nighttime UAV Tracking

Unsupervised Domain Adaptation for Nighttime Aerial Tracking

DaDiff: Domain-aware Diffusion Model for Nighttime UAV Tracking

SAM-Adapter: Adapting Segment Anything in Underperformed Scenes

Tracker Meets Night: A Transformer Enhancer for UAV Tracking

All-Day Object Tracking for Unmanned Aerial Vehicle

Scale-Aware Domain Adaptation for Robust UAV Tracking

SAM Fails to Segment Anything? – SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More

SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory

Mutual-Learning Knowledge Distillation for Nighttime UAV Tracking

SAM4UDASS: When SAM Meets Unsupervised Domain Adaptive Semantic Segmentation in Intelligent Vehicles

Enhancing Nighttime UAV Tracking with Light Distribution Suppression

SGDViT: Saliency-Guided Dynamic Vision Transformer for UAV Tracking

Unsupervised Nighttime Object Tracking Based on Transformer and Domain Adaptation Fusion Network

Joint-Optimized Unsupervised Adversarial Domain Adaptation in Remote Sensing Segmentation with Prompted Foundation Model

DCPT: Darkness Clue-Prompted Tracking in Nighttime UAVs

ADTrack: Target-Aware Dual Filter Learning for Real-Time Anti-Dark UAV Tracking

Learning to Adapt SAM for Segmenting Cross-domain Point Clouds

The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot

Modified Siamese Network Based on Feature Enhancement and Dynamic Template for Low-Light Object Tracking in UAV Videos