Progressive Scale-aware Network for Remote sensing Image Change Captioning

Chenyang Liu,Jiajun Yang,Zipeng Qi,Zhengxia Zou,Zhenwei Shi

2023-12-18

Abstract:Remote sensing (RS) images contain numerous objects of different scales, which poses significant challenges for the RS image change captioning (RSICC) task to identify visual changes of interest in complex scenes and describe them via language. However, current methods still have some weaknesses in sufficiently extracting and utilizing multi-scale information. In this paper, we propose a progressive scale-aware network (PSNet) to address the problem. PSNet is a pure Transformer-based model. To sufficiently extract multi-scale visual features, multiple progressive difference perception (PDP) layers are stacked to progressively exploit the differencing features of bitemporal features. To sufficiently utilize the extracted multi-scale features for captioning, we propose a scale-aware reinforcement (SR) module and combine it with the Transformer decoding layer to progressively utilize the features from different PDP layers. Experiments show that the PDP layer and SR module are effective and our PSNet outperforms previous methods. Our code is public at <a class="link-external link-https" href="https://github.com/Chen-Yang-Liu/PSNet" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper aims to solve the problem of multi - scale information extraction and utilization in the Remote Sensing Image Change Captioning (RSICC) task. Specifically, remote sensing images contain many objects at different scales, which brings significant challenges to identifying visual changes in complex scenes and describing these changes in natural language. Current methods still have deficiencies in fully extracting and utilizing multi - scale information.

To solve this problem, the author proposes a Progressive Scale - aware Network (PSNet) based on Transformer. PSNet is improved in the following two aspects:

1. **Extraction of multi - scale information**: PSNet uses multiple Progressive Difference Perception (PDP) layers to gradually extract the differences between bi - temporal features, thereby focusing on the changed regions and identifying visual changes. These PDP layers are stacked together to fully extract multi - scale visual features.

2. **Utilization of multi - scale information**: In order to fully utilize the extracted multi - scale features for description generation, the author proposes a Scale - aware Reinforcement (SR) module and enhances the multi - scale features from different PDP layers through a multi - head cross - attention mechanism. Multiple SR modules are combined with Transformer decoding layers to gradually utilize these multi - scale features to generate descriptions.

Experimental results show that PSNet is superior to previous methods in performance, especially in handling change descriptions of objects at different scales.