Abstract:To address the problems in the existing video super-resolution methods, such as noise, over smooth and visual artifacts, which are caused by the reliance on limited external training or mismatch of internal similarity patch instances, this study proposes a novel video super-resolution reconstruction algorithm based on deep learning and spatio-temporal feature similarity (DLSS-VSR). The video super-resolution reconstruction mechanism with the joint internal and external constraints is established utilizing the complementary advantages of both external deep correlation mapping learning and internal spatio-temporal nonlocal self-similarity prior constraint. A deep learning model based on deep convolutional neural network is constructed to learn the nonlinear correlation mapping between low-resolution and high-resolution video frame patches. A novel spatio-temporal feature similarity calculation method is proposed, which considers both internal video spatio-temporal self-similarity and external clean nonlocal similarity. For the internal spatio-temporal feature self-similarity, we improve the accuracy and robustness of similarity matching by proposing a similarity measure strategy based on spatio-temporal moment feature similarity and structural similarity. The external nonlocal similarity prior constraint is learned by the patch group-based Gaussian mixture model. The time efficiency for spatio-temporal similarity matching is further improved based on saliency detection and region correlation judgment strategy, which achieves a better tradeoff between super-resolution accuracy and speed. Experimental results demonstrate that the DLSS-VSR algorithm achieves competitive super-resolution quality compared to other state-of-the-art algorithms in both subjective and objective evaluations.

Self-tuned Deep Super Resolution

Single-image Super-Resolution Via Joint Statistic Models-Guided Deep Auto-Encoder Network

Single Remote Sensing Image Super-Resolution Via a Generative Adversarial Network with Stratified Dense Sampling and Chain Training

Orientation-Aware Deep Neural Network for Real Image Super-Resolution.

Super-Resolution for Noisy Images Via Deep Convolutional Neural Network

Image Super‐resolution Based on Self‐similarity Generative Adversarial Networks

Exploiting Self-Supervised Constraints in Image Super-Resolution

Remote Sensing Single-Image Resolution Improvement Using A Deep Gradient-Aware Network with Image-Specific Enhancement

RSAMSR: A deep neural network based on residual self-encoding and attention mechanism for image super-resolution

Coupled Deep Autoencoder for Single Image Super-Resolution.

Learning Super-Resolution Jointly from External and Internal Examples

A Two-Stage Convolutional Neural Network for Joint Demosaicking and Super-Resolution

Image Super-Resolution Using Deep Convolutional Networks

Video Super-Resolution Reconstruction Based on Deep Learning and Spatio-Temporal Feature Self-similarity

DDet: Dual-path Dynamic Enhancement Network for Real-World Image Super-Resolution

Joint Learning of Super-Resolution and Perceptual Image Enhancement for Single Image.

End-to-End Image Super-Resolution via Deep and Shallow Convolutional Networks

Unsupervised Real-World Image Super-Resolution via Dual Synthetic-to-Realistic and Realistic-to-Synthetic Translations

Self-supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations.

A Data Enhancement Method toward Real-World Single Image Super-Resolution

Deep SR-HDR: Joint Learning of Super-Resolution and High Dynamic Range Imaging for Dynamic Scenes