Abstract:Conventional methods for PAN-sharpening often struggle to restore fine details due to limitations in leveraging high-frequency information. Moreover, diffusion-based approaches lack sufficient conditioning to fully utilize Panchromatic (PAN) images and low-resolution multispectral (LRMS) inputs effectively. To address these challenges, we propose an uncertainty-aware knowledge distillation diffusion framework with details enhancement for PAN-sharpening, called U-Know-DiffPAN. The U-Know-DiffPAN incorporates uncertainty-aware knowledge distillation for effective transfer of feature details from our teacher model to a student one. The teacher model in our U-Know-DiffPAN captures frequency details through freqeuncy selective attention, facilitating accurate reverse process learning. By conditioning the encoder on compact vector representations of PAN and LRMS and the decoder on Wavelet transforms, we enable rich frequency utilization. So, the high-capacity teacher model distills frequency-rich features into a lightweight student model aided by an uncertainty map. From this, the teacher model can guide the student model to focus on difficult image regions for PAN-sharpening via the usage of the uncertainty map. Extensive experiments on diverse datasets demonstrate the robustness and superior performance of our U-Know-DiffPAN over very recent state-of-the-art PAN-sharpening methods.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve several key challenges encountered in PAN - sharpening technology, specifically including: 1. **Insufficient detail recovery**: When traditional methods fuse high - resolution panchromatic (PAN) images and low - resolution multispectral (LRMS) images, they often struggle to recover fine - grained high - frequency details. This is because these methods cannot fully utilize high - frequency information. 2. **Limitations of diffusion models**: Although methods based on diffusion models perform well, their conditioning strategies are too simplistic and fail to fully utilize the complementary information between PAN and LRMS inputs, resulting in poor restoration quality. Moreover, diffusion models are usually computationally expensive and have long inference times, which limit their practical applications. 3. **Weaknesses in knowledge distillation**: Traditional knowledge distillation methods perform poorly when handling tasks that require capturing complex details, such as super - resolution, because they learn all regions equally without paying special attention to regions that require concentrated learning. To solve these problems, the paper proposes a new framework - U - Know - DiffPAN, which combines uncertainty - aware knowledge distillation (Uncertainty - aware Knowledge Distillation, U - Know) and a detail - enhancement strategy to improve the effectiveness of PAN - sharpening. Specifically: - **Uncertainty - aware knowledge distillation**: By introducing an uncertainty map, the student model is guided to focus more on difficult areas that are difficult for the teacher model to restore. - **Frequency - selective attention**: Utilize the frequency - selective attention mechanism (Frequency Selective Attention, FSA) to extract and enhance frequency information from PAN and LRMS inputs. - **Efficient encoding and decoding structures**: Employ techniques such as compact vector representation and wavelet transform to ensure the effective utilization of frequency information. Through these improvements, U - Know - DiffPAN can generate more detailed and robust high - resolution multispectral (HRMS) images while maintaining a lower computational cost, especially performing excellently in areas of high uncertainty. ### Formula summary 1. **Residual prediction formula**: \[ X_0 = I_{\text{HRMS}}-I_{\text{LRMS}} \] 2. **Forward diffusion process**: \[ X_t=\sqrt{\bar{\alpha}_t}X_0+\sqrt{1 - \bar{\alpha}_t}\epsilon \] where \(\epsilon\sim\mathcal{N}(0, I)\), \(\bar{\alpha}_t = \prod_{s = 1}^t(1-\beta_s)\). 3. **Uncertainty - driven diffusion loss function**: \[ L_{\text{U - Diff}}=\frac{1}{2}\hat{\theta}\odot\|\hat{X}_0 - X_0\|^2+\frac{1}{2}\log\hat{\theta} \] 4. **U - Know loss function**: \[ L_{\text{U - Know}}=L_{\text{hard}}+\lambda_sL_{\text{soft}}+\lambda_fL_{\text{feat}} \] where, \[ L_{\text{hard}}=(\tau+\hat{\theta})\odot\|\hat{X}_0 - X_0\| \] \[ L_{\text{soft}}=(\tau-\hat{\theta})\odot\|\hat{X}_0

U-Know-DiffPAN: An Uncertainty-aware Knowledge Distillation Diffusion Framework with Details Enhancement for PAN-Sharpening