ScoreSeg: Leveraging Score-Based Generative Model for Self-Supervised Semantic Segmentation of Remote Sensing

Junzhe Lu,Guangjun He,Hongkun Dou,Qing Gao,Leyuan Fang,Yue Deng
DOI: https://doi.org/10.1109/jstars.2023.3314866
IF: 4.715
2023-10-04
IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Abstract:The performance of semantic segmentation of remote sensing images (RSIs) heavily depends on the number of pixel-level annotations. In practice, the accumulation of pixel-level annotations for large RSIs is quite expensive or even impossible under certain scenarios. Here, we try to solve this data-intensive problem from the novel aspect of score-based self-supervise learning (SSL) and introduce a robust RSI semantic segmentation model called ScoreSeg. Unlike traditional pixel-level SSL paradigms, the generative SSL mechanism in ScoreSeg is simple in loss design and stable in pretraining, granting it an indispensable ability in dense feature learning from very large RSIs. In the model implementation, ScoreSeg first extracts pixelwise representations of RSIs by pretraining a time-dependent score-based model on abundant off-the-shelf unlabeled RSIs. Then, to address the sparse feature problem in RSIs, the collected features from different timesteps and resolutions are aggregated together forming a rich feature map for downstream semantic segmentation. Experimental results on three datasets show that our proposed ScoreSeg outperforms state-of-the-art (SOTA) SSL methods and alternative pretraining models on ImageNet by nontrivial margins, especially with very limited annotations.
imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geography, physical
What problem does this paper attempt to address?