PNEN: Pyramid Non-Local Enhanced Networks

Feida Zhu,Chaowei Fang,Kai-Kuang Ma

DOI: https://doi.org/10.1109/TIP.2020.3019644

2020-08-22

Abstract:Existing neural networks proposed for low-level image processing tasks are usually implemented by stacking convolution layers with limited kernel size. Every convolution layer merely involves in context information from a small local neighborhood. More contextual features can be explored as more convolution layers are adopted. However it is difficult and costly to take full advantage of long-range dependencies. We propose a novel non-local module, Pyramid Non-local Block, to build up connection between every pixel and all remain pixels. The proposed module is capable of efficiently exploiting pairwise dependencies between different scales of low-level structures. The target is fulfilled through first learning a query feature map with full resolution and a pyramid of reference feature maps with downscaled resolutions. Then correlations with multi-scale reference features are exploited for enhancing pixel-level feature representation. The calculation procedure is economical considering memory consumption and computational cost. Based on the proposed module, we devise a Pyramid Non-local Enhanced Networks for edge-preserving image smoothing which achieves state-of-the-art performance in imitating three classical image smoothing algorithms. Additionally, the pyramid non-local block can be directly incorporated into convolution neural networks for other image restoration tasks. We integrate it into two existing methods for image denoising and single image super-resolution, achieving consistently improved performance.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the effective utilization of long - distance dependencies in low - level image processing tasks. When dealing with such tasks, existing neural networks are usually implemented by stacking convolutional layers with limited kernel sizes, and each convolutional layer only involves context information from a small local neighborhood. Although adding more convolutional layers can explore more context features, it is both difficult and expensive to fully utilize long - distance dependencies. For this reason, the author proposes a new non - local module - Pyramid Non - local Block (PNB), which aims to establish connections between each pixel and all other pixels, thereby efficiently mining pairwise dependencies between low - level structures at different scales. Based on this module, the author designs Pyramid Non - local Enhanced Networks (PNEN) for edge - preserving image smoothing and achieves state - of - the - art performance in imitating three classic image smoothing algorithms. In addition, the Pyramid Non - local Block can also be directly integrated into convolutional neural networks for other image restoration tasks, such as image denoising and single - image super - resolution, achieving continuously improved performance.