Abstract:Deep neural networks for image super-resolution (ISR) have shown significant advantages over traditional approaches like the interpolation. However, they are often criticized as 'black boxes' compared to traditional approaches with solid mathematical foundations. In this paper, we attempt to interpret the behavior of deep neural networks in ISR using theories from the field of signal processing. First, we report an intriguing phenomenon, referred to as `the sinc phenomenon.' It occurs when an impulse input is fed to a neural network. Then, building on this observation, we propose a method named Hybrid Response Analysis (HyRA) to analyze the behavior of neural networks in ISR tasks. Specifically, HyRA decomposes a neural network into a parallel connection of a linear system and a non-linear system and demonstrates that the linear system functions as a low-pass filter while the non-linear system injects high-frequency information. Finally, to quantify the injected high-frequency information, we introduce a metric for image-to-image tasks called Frequency Spectrum Distribution Similarity (FSDS). FSDS reflects the distribution similarity of different frequency components and can capture nuances that traditional metrics may overlook. Code, videos and raw experimental results for this paper can be found in:

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the "black - box" nature of deep neural networks in the Image Super - Resolution (ISR) task. That is, although these networks have shown more significant advantages than traditional methods, their internal mechanisms are still unclear. The paper explains the behavior of deep neural networks in ISR through signal processing theory, especially exploring whether neural networks work like traditional low - pass filters and how to quantify the high - frequency information injected by neural networks. Specifically, the paper mainly solves the following problems: 1. **Explore the low - pass filtering behavior of neural networks**: The paper finds that when an impulse input is sent into a neural network, the output of the network exhibits a phenomenon similar to the sinc function, which the author calls the "sinc phenomenon". This phenomenon indicates that neural networks can simulate the behavior of low - pass filters to some extent. 2. **Propose Hybrid Response Analysis (HyRA)**: Based on the above findings, the paper proposes the HyRA method, which decomposes a neural network into a parallel combination of a linear system and a nonlinear system. Among them, the linear system acts as a low - pass filter, while the nonlinear system is responsible for injecting high - frequency information. 3. **Introduce the Frequency Spectrum Distribution Similarity (FSDS) index**: In order to quantitatively describe the high - frequency information injected by neural networks, the paper proposes a new evaluation index - FSDS. This index is based on the power distribution map of the spectrum and can capture detailed changes that may be ignored by traditional evaluation indexes (such as PSNR, SSIM). By solving these problems, the paper not only deepens the understanding of the behavior of deep neural networks in ISR tasks but also provides a new tool to evaluate and improve the performance of ISR networks.

Exploring the Low-Pass Filtering Behavior in Image Super-Resolution

FreqNet: A Frequency-domain Image Super-Resolution Network with Dicrete Cosine Transform

HIIF: Hierarchical Encoding based Implicit Image Function for Continuous Super-resolution

Multi-level feature interaction image super-resolution network based on convolutional nonlinear spiking neural model

Fast and Accurate Single Image Super-Resolution Via an Energy-Aware Improved Deep Residual Network.

SFHN: Spatial-Frequency Domain Hybrid Network for Image Super-Resolution

Single Image Super-Resolution Neural Network Using Frequency-Domain Information

Iterative Network for Image Super-Resolution

Orientation-Aware Deep Neural Network for Real Image Super-Resolution.

Learning Frequency-aware Dynamic Network for Efficient Super-Resolution.

A High-Frequency Focused Network for Lightweight Single Image Super-Resolution

Hyperspectral Image Super-Resolution via Dual-domain Network Based on Hybrid Convolution

Deeply Recursive Low- and High-Frequency Fusing Networks for Single Image Super-Resolution

Hierarchical Similarity Learning for Aliasing Suppression Image Super-Resolution

A Lightweight Image Super-Resolution Reconstruction Algorithm Based on the Residual Feature Distillation Mechanism

Subspace Constraint for Single Image Super-Resolution

A Closed-Loop Network for Single Infrared Remote Sensing Image Super-Resolution in Real World

An Attention-Based Approach for Single Image Super Resolution.

Two-Stage Spatial-Frequency Joint Learning for Large-Factor Remote Sensing Image Super-Resolution

Faster Image Super-Resolution by Improved Frequency-Domain Neural Networks.

Research on Image Super-Resolution Reconstruction Mechanism based on Convolutional Neural Network