MT-HCCAR: Multi-Task Deep Learning with Hierarchical Classification and Attention-based Regression for Cloud Property Retrieval

Xingyan Li,Andrew M. Sayer,Ian T. Carroll,Xin Huang,Jianwu Wang

2024-07-05

Abstract:In the realm of Earth science, effective cloud property retrieval, encompassing cloud masking, cloud phase classification, and cloud optical thickness (COT) prediction, remains pivotal. Traditional methodologies necessitate distinct models for each sensor instrument due to their unique spectral characteristics. Recent strides in Earth Science research have embraced machine learning and deep learning techniques to extract features from satellite datasets' spectral observations. However, prevailing approaches lack novel architectures accounting for hierarchical relationships among retrieval tasks. Moreover, considering the spectral diversity among existing sensors, the development of models with robust generalization capabilities over different sensor datasets is imperative. Surprisingly, there is a dearth of methodologies addressing the selection of an optimal model for diverse datasets. In response, this paper introduces MT-HCCAR, an end-to-end deep learning model employing multi-task learning to simultaneously tackle cloud masking, cloud phase retrieval (classification tasks), and COT prediction (a regression task). The MT-HCCAR integrates a hierarchical classification network (HC) and a classification-assisted attention-based regression network (CAR), enhancing precision and robustness in cloud labeling and COT prediction. Additionally, a comprehensive model selection method rooted in K-fold cross-validation, one standard error rule, and two introduced performance scores is proposed to select the optimal model over three simulated satellite datasets OCI, VIIRS, and ABI. The experiments comparing MT-HCCAR with baseline methods, the ablation studies, and the model selection affirm the superiority and the generalization capabilities of MT-HCCAR.

Machine Learning,Computer Vision and Pattern Recognition,Signal Processing

What problem does this paper attempt to address?

The paper attempts to address the problem of accurately extracting cloud properties from satellite data in Earth sciences, including cloud masking (distinguishing between cloudy and cloud-free pixels), cloud phase classification (determining whether clouds are composed of liquid water droplets or ice crystals), and cloud optical thickness (COT, reflecting the amount of light scattering by clouds and their internal liquid or ice content) prediction. Traditional methods rely on different models tailored to the unique spectral characteristics of each sensor, leading to poor model generalization and insufficient cross-sensor method evaluation. Additionally, existing methods lack architectural designs capable of capturing hierarchical relationships between tasks. To this end, the paper proposes an end-to-end multi-task deep learning model, MT-HCCAR, which simultaneously handles cloud masking, cloud phase classification (classification task), and COT prediction (regression task) through multi-task learning. MT-HCCAR integrates a hierarchical classification network (HC) and a classification-assisted attention regression network (CAR) to improve the accuracy and robustness of cloud labeling and COT prediction. Experimental evaluations show that MT-HCCAR performs excellently across various evaluation metrics and satellite datasets, outperforming baseline methods.

MT-HCCAR: Multi-Task Deep Learning with Hierarchical Classification and Attention-based Regression for Cloud Property Retrieval

Multi-Task Collaboration Deep Learning Framework for Infrared Precipitation Estimation.

Transfer-Learning-Based Approach to Retrieve the Cloud Properties Using Diverse Remote Sensing Datasets

Multi-temporal Cloud Detection Based on Robust PCA for Optical Remote Sensing Imagery

CMAAC: Combining Multiattention and Asymmetric Convolution Global Learning Framework for Hyperspectral Image Classification

Discriminative Feature Learning Constrained Unsupervised Network for Cloud Detection in Remote Sensing Imagery.

Deep learning based cloud detection for medium and high resolution remote sensing images of different sensors

Research on infrared hyperspectral remote sensing cloud detection method based on deep learning

AMCD-Net: An Effective Attention-Aided Multilevel Cloud Detection Network for Optical Satellite Imagery

Physics-driven machine learning algorithm facilitates multilayer cloud property retrievals from geostationary passive imager measurements

A multi-task driven and reconfigurable network for cloud detection in cloud-snow coexistence regions from very-high-resolution remote sensing images

C-LiSA: A Hierarchical Hybrid Transformer Model Using Orthogonal Cross Attention for Satellite Image Cloud Segmentation

PMAA: A Progressive Multi-scale Attention Autoencoder Model for High-performance Cloud Removal from Multi-temporal Satellite Imagery

MSMT-LCL: Multiscale Spatial-Spectral Masked Transformer With Local Contrastive Learning for Hyperspectral Image Classification

Attentive Contextual Attention for Cloud Removal

Joint Classification of Hyperspectral and LiDAR Data Based on Position-Channel Cooperative Attention Network

CLiSA: A Hierarchical Hybrid Transformer Model using Orthogonal Cross Attention for Satellite Image Cloud Segmentation

A New SCAE-MT Classification Model for Hyperspectral Remote Sensing Images

SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal

Hyperspectral image-aided LiDAR point cloud labeling via spatio-spectral feature representation learning

Unpaired Thick Cloud Removal with Multiscale SAR-optical Data Fusion