Transfer Learning with Pretrained Remote Sensing Transformers

Anthony Fuller,Koreen Millard,James R. Green

DOI: https://doi.org/10.48550/arXiv.2209.14969

2022-09-29

Abstract:Although the remote sensing (RS) community has begun to pretrain transformers (intended to be fine-tuned on RS tasks), it is unclear how these models perform under distribution shifts. Here, we pretrain a new RS transformer--called SatViT-V2--on 1.3 million satellite-derived RS images, then fine-tune it (along with five other models) to investigate how it performs on distributions not seen during training. We split an expertly labeled land cover dataset into 14 datasets based on source biome. We train each model on each biome separately and test them on all other biomes. In all, this amounts to 1638 biome transfer experiments. After fine-tuning, we find that SatViT-V2 outperforms SatViT-V1 by 3.1% on in-distribution (matching biomes) and 2.8% on out-of-distribution (mismatching biomes) data. Additionally, we find that initializing fine-tuning from the linear probed solution (i.e., leveraging LPFT [1]) improves SatViT-V2's performance by another 1.2% on in-distribution and 2.4% on out-of-distribution data. Next, we find that pretrained RS transformers are better calibrated under distribution shifts than non-pretrained models and leveraging LPFT results in further improvements in model calibration. Lastly, we find that five measures of distribution shift are moderately correlated with biome transfer performance. We share code and pretrained model weights. (<a class="link-external link-https" href="https://github.com/antofuller/SatViT" rel="external noopener nofollow">this https URL</a>)

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are the performance of pre - trained Transformers in Remote Sensing (RS) under Distribution Shifts and their calibration capabilities. Specifically, the researchers focus on the following points:

1. **The impact of pre - training on performance**: The researchers pre - trained a new Remote Sensing Transformer model - SatViT - V2 and explored its transfer learning ability between different biomes. In particular, they hope to understand whether pre - training can improve the performance of the model on unseen distributions.

2. **The effect of Linear Probe Fine - Tuning (LPFT)**: The researchers also explored the effect of using the Linear Probe Fine - Tuning method to fine - tune the pre - trained model. Can this method improve the performance of the model under Distribution Shifts while maintaining or improving its performance in calibration?

3. **Model calibration capabilities**: The researchers evaluated whether the predicted probabilities of the pre - trained model can accurately reflect the uncertainty of its predictions during transfer learning between different biomes. This involves the calibration capabilities of the model, that is, whether the probabilities predicted by the model are consistent with the actually correct probabilities.

4. **Measurement of Distribution Shifts**: The researchers proposed five methods for measuring Distribution Shifts and explored the correlation between these measurements and the performance of the model in transfer learning between different biomes. This helps to predict the performance of the model on the target data set.

Through these studies, the authors hope to provide valuable insights for researchers and practitioners in the field of Remote Sensing, especially on how to better understand and apply transfer learning techniques when using large pre - trained Transformer models.