Abstract:The nonlocal self-similarity in natural image provides an effective prior for single image super-resolution (SISR), which is beneficial to contextual information capture and performance improvement, as demonstrated by conventional SISR methods. However, it is little explored to utilize this property in deep neural networks. In this paper, we propose a self-similarity prior guided (SSPG) network to incorporate self-similarity-based nonlocal operation into deep neural network for SISR. Specifically, we design a cross-scale nearest-neighbor residual (CSNNR) block via introducing cross-scale <span class="mjpage"><svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1.211ex" height="2.176ex" style="vertical-align: -0.338ex;" viewBox="0 -791.3 521.5 936.9" role="img" focusable="false" xmlns="http://www.w3.org/2000/svg"><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"> <use xlink:href="#MJMATHI-6B" x="0" y="0"></use></g></svg></span> -nearest neighbors (KNN) matching into a residual block, which can be flexibly integrated into deep networks to capture long-range correlations among multi-scale and multi-level features. Meanwhile, by stacking a CSNNR block and a sequence of wide-activated residual blocks with a local skip-connection, a multi-level residual self-similarity (MRSS) module is developed to effectively employ local and nonlocal information for detail recovery. Thus, through cascading multiple MRSS modules, the proposed SSPG network performs both self-similarity-based nonlocal operation and convolution-based local operation on multi-level features to reconstruct informative features for accurate SISR. In addition, for pursuing visually pleasing results, we apply our SSPG network to the perception-oriented SISR field by following the framework of generative adversarial networks. In particular, we explore a sample-discriminating learning mechanism based on the statistical descriptions of training samples, and include it in optimization procedure to automatically tune the contributions of different samples according to their characteristics and then focus the network on creating realistic results. Extensive quantitative and qualitative evaluations on benchmark datasets illustrate the superiority of our proposed models over the stat--of-the-art methods for both distortion-oriented and perception-oriented image super-resolution tasks.<svg xmlns="http://www.w3.org/2000/svg" style="display: none;"><defs id="MathJax_SVG_glyphs"><path stroke-width="1" id="MJMATHI-6B" d="M121 647Q121 657 125 670T137 683Q138 683 209 688T282 694Q294 694 294 686Q294 679 244 477Q194 279 194 272Q213 282 223 291Q247 309 292 354T362 415Q402 442 438 442Q468 442 485 423T503 369Q503 344 496 327T477 302T456 291T438 288Q418 288 406 299T394 328Q394 353 410 369T442 390L458 393Q446 405 434 405H430Q398 402 367 380T294 316T228 255Q230 254 243 252T267 246T293 238T320 224T342 206T359 180T365 147Q365 130 360 106T354 66Q354 26 381 26Q429 26 459 145Q461 153 479 153H483Q499 153 499 144Q499 139 496 130Q455 -11 378 -11Q333 -11 305 15T277 90Q277 108 280 121T283 145Q283 167 269 183T234 206T200 217T182 220H180Q168 178 159 139T145 81T136 44T129 20T122 7T111 -2Q98 -11 83 -11Q66 -11 57 -1T48 16Q48 26 85 176T158 471L195 616Q196 629 188 632T149 637H144Q134 637 131 637T124 640T121 647Z"></path></defs></svg>

Self-Learning Super-Resolution Using Convolutional Principal Component Analysis and Random Matching

Single-Image Superresolution Based on Local Regression and Nonlocal Self-Similarity

Single Image Super-Resolution with Multiscale Similarity Learning.

SERF: A Simple, Effective, Robust, and Fast Image Super-Resolver from Cascaded Linear Regression.

Learning Cascaded Convolutional Networks for Blind Single Image Super-Resolution

Single image super-resolution using feature adaptive learning and global structure sparsity

Video Super-Resolution Reconstruction Based on Deep Learning and Spatio-Temporal Feature Self-similarity

Single Image Super-Resolution Via Locally Regularized Anchored Neighborhood Regression and Nonlocal Means.

A self-learning image super-resolution method via sparse representation and non-local similarity

Position Constraint Based Face Image Super-Resolution by Learning Multiple Local Linear Projections

Convolutional Sparse Coding for Image Super-Resolution

Learning-based Super-Resolution Method with a Combining of Both Global and Local Constraints

Single-image Super-resolution Algorithm Based on Multi-scale Nonlocal Regularization

Learning Local Gaussian Process Regression for Image Super-Resolution.

Exploiting Self-Supervised Constraints in Image Super-Resolution

Resolution Enhancement Based on Learning the Sparse Association of Image Patches

Self-attention learning network for face super-resolution

Local Patch Classification Based Framework for Single Image Super-Resolution.

Single Image Super Resolution Reconstruction Algorithm Based on Deep Learning

Image Super-resolution Based on Multi-scale Self-similarity and Gradient Constraint

Image Super-Resolution With Self-Similarity Prior Guided Network and Sample-Discriminating Learning