A Multispectral Remote Sensing Crop Segmentation Method Based on Segment Anything Model Using Multi-stage Adaptation Fine-tuning

Binbin Song,Hui Yang,Yanlan Wu,Peng Zhang,Biao Wang,Guichao Han

DOI: https://doi.org/10.1109/tgrs.2024.3411398

IF: 8.2

2024-01-01

IEEE Transactions on Geoscience and Remote Sensing

Abstract:Multispectral information is crucial for remote sensing crop monitoring, but current methods struggle with inadequate feature extraction, leading to poor generalization and incomplete segmentation. The segment anything model (SAM) shows significant potential for generalization across fields, offering a promising solution for crop monitoring. This article introduces a crop segmentation method based on SAM using multistage adaptation fine-tuning, namely MAF-SAM, effectively utilizing information from multispectral remote sensing and the transfer and generalization abilities of SAM. In its first stage, MAF-SAM employs a prefix adapter to extract primary low-level multispectral features and compresses them into three channels to meet the requirements of subsequent stages. The second stage introduces a low-rank adaptation (LoRA) fine-tuning strategy to inject crop-specific knowledge into the image encoder, enhancing MAF-SAM's adaptability in particular crop segmentation tasks. In the third stage, it utilizes a mask decoder with no-prompt embedding to automatically generate masks with accurate class information. MAF-SAM achieves F1 scores and Intersection over Union (IoU) for soybean and corn of 0.9294, 0.8680, 0.8760, and 0.7723, respectively, along with a Kappa coefficient of 0.9543. It demonstrates superior temporal and spatial transfer capabilities relative to five other advanced segmentation methods in our study area.