Abstract:Video super-resolution (VSR) is to restore a photo-realistic high-resolution (HR) frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames). An important step in VSR is to fuse the feature of the reference frame with the features of the supporting frames. The major issue with existing VSR methods is that the fusion is conducted in a one-stage manner, and the fused feature may deviate greatly from the visual information in the original LR reference frame. In this paper, we propose an end-to-end Multi-Stage Feature Fusion Network that fuses the temporally aligned features of the supporting frames and the spatial feature of the original reference frame at different stages of a feed-forward neural network architecture. In our network, the Temporal Alignment Branch is designed as an inter-frame temporal alignment module used to mitigate the misalignment between the supporting frames and the reference frame. Specifically, we apply the multi-scale dilated deformable convolution as the basic operation to generate temporally aligned features of the supporting frames. Afterwards, the Modulative Feature Fusion Branch, the other branch of our network accepts the temporally aligned feature map as a conditional input and modulates the feature of the reference frame at different stages of the branch backbone. This enables the feature of the reference frame to be referenced at each stage of the feature fusion process, leading to an enhanced feature from LR to HR. Experimental results on several benchmark datasets demonstrate that our proposed method can achieve state-of-the-art performance on VSR task.

Recursive Multi-Stage Upscaling Network with Discriminative Fusion for Super-Resolution

Multi-Scale Cross-Attention Fusion Network Based on Image Super-Resolution

Lightweight Super-Resolution Using Deep Neural Learning

A deep recursive multi-scale feature fusion network for image super-resolution?

Deep multi-level up-projection network for single image super-resolution

Lightweight Multiresolution Feature Fusion Network for Spectral Super-Resolution

Single-image Super-Resolution Via Selective Multi-Scale Network

A Multi-Scale Recursive Attention Feature Fusion Network for Image Super-Resolution Reconstruction Algorithm

Multi-Level Residual Up-Projection Activation Network For Image Super-Resolution

Channel Attention and Multi-level Features Fusion for Single Image Super-Resolution

Multi-Scale Residual Network For Image Super-Resolution

Deeply Recursive Low- and High-Frequency Fusing Networks for Single Image Super-Resolution

Image Super-Resolution Via a Novel Cascaded Convolutional Neural Network Framework.

Multiscale Residual Dense Network for the Super-Resolution of Remote Sensing Images

Multi-Level Feature Fusion Mechanism for Single Image Super-Resolution

Dual-Branch Multiscale Channel Fusion Unfolding Network for Optical Remote Sensing Image Super-Resolution.

An Image Super-Resolution Network Based on Multi-Scale Convolution Fusion

Gradient-Guided and Multi-Scale Feature Network for Image Super-Resolution

Multi-Stage Feature Fusion Network for Video Super-Resolution

Multi-scale feature aggregation network for Image super-resolution

Face Super-Resolution Via Hierarchical Multi-Scale Residual Fusion Network.