RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

Fan Liu,Delong Chen,Zhangqingyun Guan,Xiaocong Zhou,Jiale Zhu,Qiaolin Ye,Liyong Fu,Jun Zhou

2024-04-16

Abstract:General-purpose foundation models have led to recent breakthroughs in artificial intelligence. In remote sensing, self-supervised learning (SSL) and Masked Image Modeling (MIM) have been adopted to build foundation models. However, these models primarily learn low-level features and require annotated data for fine-tuning. Moreover, they are inapplicable for retrieval and zero-shot applications due to the lack of language understanding. To address these limitations, we propose RemoteCLIP, the first vision-language foundation model for remote sensing that aims to learn robust visual features with rich semantics and aligned text embeddings for seamless downstream application. To address the scarcity of pre-training data, we leverage data scaling which converts heterogeneous annotations into a unified image-caption data format based on Box-to-Caption (B2C) and Mask-to-Box (M2B) conversion. By further incorporating UAV imagery, we produce a 12 $\times$ larger pretraining dataset than the combination of all available datasets. RemoteCLIP can be applied to a variety of downstream tasks, including zero-shot image classification, linear probing, $\textit{k}$-NN classification, few-shot classification, image-text retrieval, and object counting in remote sensing images. Evaluation on 16 datasets, including a newly introduced RemoteCount benchmark to test the object counting ability, shows that RemoteCLIP consistently outperforms baseline foundation models across different model scales. Impressively, RemoteCLIP beats the state-of-the-art method by 9.14% mean recall on the RSITMD dataset and 8.92% on the RSICD dataset. For zero-shot classification, our RemoteCLIP outperforms the CLIP baseline by up to 6.39% average accuracy on 12 downstream datasets. Project website: <a class="link-external link-https" href="https://github.com/ChenDelong1999/RemoteCLIP" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address several key limitations of existing foundation models in the field of remote sensing. Specifically:

1. **Low-level Feature Learning**: Existing foundation models based on Self-Supervised Learning (SSL) and Masked Image Modeling (MIM) primarily learn low-level features and lack rich semantic information.
2. **Data Annotation Requirement**: These models require annotated data for fine-tuning and do not perform well in zero-shot tasks because they lack language understanding capabilities.
3. **Data Scarcity**: Pre-training data in the remote sensing field is very limited, which restricts the learning ability and generalization performance of the models.

To address these issues, the authors propose **RemoteCLIP**, the first vision-language foundation model specifically for the remote sensing field. The goal of RemoteCLIP is to learn robust visual features with rich semantics and generate text embeddings aligned with visual features, enabling seamless downstream applications.

### Solutions

1. **Data Augmentation**: To tackle the scarcity of pre-training data, the authors convert heterogeneous annotations into a unified image-caption data format through data augmentation. Specific methods include:
- **Box-to-Caption (B2C)**: Converting object detection bounding box annotations into natural language descriptions.
- **Mask-to-Box (M2B)**: Converting mask annotations in segmentation datasets into bounding box annotations, and further into caption data.
- **Drone Imagery**: Introducing drone imagery to enhance the diversity of pre-training data.

2. **Model Training**: By optimizing the InfoNCE loss function, the visual and language representations are aligned. The pre-trained RemoteCLIP can be applied to various downstream tasks, including zero-shot image classification, linear probing, k-NN classification, few-shot classification, image-text retrieval, and object counting.

### Experimental Results

The authors evaluated the model on 16 datasets, including the newly introduced RemoteCount benchmark to verify the model's object counting capability. Experimental results show that RemoteCLIP significantly outperforms baseline foundation models on multiple datasets. For example, on the RSITMD dataset, RemoteCLIP's average recall is 9.14% higher than the state-of-the-art method, and 8.92% higher on the RSICD dataset. In zero-shot classification tasks, RemoteCLIP's average accuracy on 12 downstream datasets is 6.39% higher than the CLIP baseline.

### Main Contributions

1. **Large-scale Remote Sensing Dataset**: Constructed a comprehensive dataset combining various remote sensing datasets, which is 12 times the size of existing datasets, addressing the issue of pre-training data scarcity.
2. **Novel Vision-Language Foundation Model**: Proposed RemoteCLIP, which learns robust visual features and rich semantic information through large-scale pre-training data.
3. **Diverse Downstream Applications**: Evaluated RemoteCLIP's performance in various downstream tasks, including cross-modal retrieval, zero-shot/few-shot/full-shot satellite image classification, and object counting, and introduced a new benchmark, RemoteCount.

Through these methods and experiments, RemoteCLIP provides strong support for vision-language tasks in the remote sensing field, demonstrating superior performance and generalization capabilities in various application scenarios.