Abstract:SIAM Journal on Imaging Sciences, Volume 17, Issue 1, Page 441-475, March 2024. High-dimensional deep features extracted by convolutional neural networks have nonlocal self-similarity. However, incorporating this nonlocal prior of deep features into deep network architectures with an interpretable variational framework is rarely explored. In this paper, we propose a learnable nonlocal self-similarity deep feature network for image denoising. Our method is motivated by the fact that the high-dimensional deep features obey a mixture probability distribution based on the Parzen–Rosenblatt window method. Then a regularizer with learnable nonlocal weights is proposed by considering the dual representation of the log-probability prior of the deep features. Specifically, the nonlocal weights are introduced as dual variables that can be learned by unrolling the associated numerical scheme. This leads to nonlocal modules (NLMs) in newly designed networks. Our method provides a statistical and variational interpretation for the nonlocal self-attention mechanism widely used in various networks. By adopting nonoverlapping window and region decomposition techniques, we can significantly reduce the computational complexity of nonlocal self-similarity, thus enabling parallel computation of the NLM. The solution to the proposed variational problem can be formulated as a learnable nonlocal self-similarity network for image denoising. This work offers a novel approach for constructing network structures that consider self-similarity and nonlocality. The improvements achieved by this method are predictable and partially controllable. Compared with several closely related denoising methods, the experimental results show the effectiveness of the proposed method in image denoising.

Higher-order Nonlocal Variational Model

Variational Image Matting with Local and Nonlocal Operators

A structure-preserved nonlocal iterative regularization model for image denoising

Nonlocal Variational Model and Filter Algorithm to Remove Multiplicative Noise

Mixed Higher Order Variational Model for Image Recovery

Two-direction Nonlocal Model for Image Interpolation

Adaptive Total-Variation and Nonconvex Low-Rank Model for Image Denoising

Research on Non-Local Regularization Model and Algorithms for Image Super-Resolution Reconstruction

Nonconvex Second-Order Variational Image Denoising Model with Adaptive Selection of Regularization Parameters

Local and Nonlocal Steering Kernel Weighted Total Variation Model for Image Denoising.

An Efficient Variational-Level-Set Model Based on Adaptive Local Fitted Image for Noisy Image Segmentation

Image Denoising Via Nonlocal Low Rank Approximation with Local Structure Preserving

Image Denoising Via Asymptotic Nonlocal Filtering

Structural Similarity-Based Nonlocal Variational Models for Image Restoration.

Learnable Nonlocal Self-Similarity of Deep Features for Image Denoising

Poisson Noise Reduction with Higher-Order Natural Image Prior Model.

Variational Model with Kernel Metric-Based Data Term for Noisy Image Segmentation

Wavelet-Based Total Variation and Nonlocal Similarity Model for Image Denoising

A Variational Formulation for Physical Noised Image Segmentation

Image recovery via nonlocal operators

Non-local Means Image Denoising in Wavelet Domain with Unknown Noise Variance