Remote Diffusion

Kunal Sunil Kasodekar

2024-05-08

Abstract:I explored adapting Stable Diffusion v1.5 for generating domain-specific satellite and aerial images in remote sensing. Recognizing the limitations of existing models like Midjourney and Stable Diffusion, trained primarily on natural RGB images and lacking context for remote sensing, I used the RSICD dataset to train a Stable Diffusion model with a loss of 0.2. I incorporated descriptive captions from the dataset for text-conditioning. Additionally, I created a synthetic dataset for a Land Use Land Classification (LULC) task, employing prompting techniques with RAG and ChatGPT and fine-tuning a specialized remote sensing LLM. However, I faced challenges with prompt quality and model performance. I trained a classification model (ResNet18) on the synthetic dataset achieving 49.48% test accuracy in TorchGeo to create a baseline. Quantitative evaluation through FID scores and qualitative feedback from domain experts assessed the realism and quality of the generated images and dataset. Despite extensive fine-tuning and dataset iterations, results indicated subpar image quality and realism, as indicated by high FID scores and domain-expert evaluation. These findings call attention to the potential of diffusion models in remote sensing while highlighting significant challenges related to insufficient pretraining data and computational resources.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem this paper attempts to address is the inadequacy of existing generative models (such as Midjourney and Stable Diffusion) in generating satellite and aerial images in specific domains. These models are primarily trained on natural RGB images and lack specific knowledge and context in the field of remote sensing, thus failing to generate high-quality, high-resolution satellite and aerial images. The authors address this issue through the following methods:

1. **Adapting the Stable Diffusion model**: The authors fine-tuned Stable Diffusion v1.5 using the RSICD dataset to generate satellite and aerial images in specific domains.
2. **Text-conditioned generation**: Utilizing descriptive sentences provided in the RSICD dataset for text-conditioned generation.
3. **Synthetic dataset creation**: For downstream tasks (such as land use and land cover classification), the authors created a synthetic dataset and used RAG and ChatGPT to generate text prompts.
4. **Model evaluation**: Evaluating the quality and authenticity of the generated images through FID scores and qualitative feedback from domain experts.

Despite multiple fine-tuning and dataset iterations, the results indicate that the quality and authenticity of the generated images are still unsatisfactory, primarily due to insufficient pre-training data and limited computational resources. These findings highlight the potential of diffusion models in the field of remote sensing while also pointing out the significant challenges they face.