Abstract:Video super-resolution (VSR) is to restore a photo-realistic high-resolution (HR) frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames). An important step in VSR is to fuse the feature of the reference frame with the features of the supporting frames. The major issue with existing VSR methods is that the fusion is conducted in a one-stage manner, and the fused feature may deviate greatly from the visual information in the original LR reference frame. In this paper, we propose an end-to-end Multi-Stage Feature Fusion Network that fuses the temporally aligned features of the supporting frames and the spatial feature of the original reference frame at different stages of a feed-forward neural network architecture. In our network, the Temporal Alignment Branch is designed as an inter-frame temporal alignment module used to mitigate the misalignment between the supporting frames and the reference frame. Specifically, we apply the multi-scale dilated deformable convolution as the basic operation to generate temporally aligned features of the supporting frames. Afterwards, the Modulative Feature Fusion Branch, the other branch of our network accepts the temporally aligned feature map as a conditional input and modulates the feature of the reference frame at different stages of the branch backbone. This enables the feature of the reference frame to be referenced at each stage of the feature fusion process, leading to an enhanced feature from LR to HR. Experimental results on several benchmark datasets demonstrate that our proposed method can achieve state-of-the-art performance on VSR task.

Single image super-resolution via deep progressive multi-scale fusion networks

Multi-Scale Cross-Attention Fusion Network Based on Image Super-Resolution

Lightweight Multi-Attention Fusion Network for Image Super-Resolution

Single-image Super-Resolution Via Selective Multi-Scale Network

A Progressive Feature Enhancement Deep Network for Large-Scale Remote Sensing Image Superresolution

Multi-Scale Progressive Fusion Learning for Depth Map Super-Resolution

Face Super-Resolution Via Hierarchical Multi-Scale Residual Fusion Network.

Single image super-resolution with multi-scale information cross-fusion network

Image super-resolution reconstruction based on attention mechanism and feature fusion

Multi-Stage Feature Fusion Network for Video Super-Resolution

Enhanced Implicit Function-Based Network for Arbitrary-Scale Image Super-Resolution

Image Super-Resolution Using Deep Convolutional Networks

Multi-Scale Residual Network For Image Super-Resolution

Remote Sensing Imagery Super Resolution Based on Adaptive Multi-Scale Feature Fusion Network

A Multi-Scale Recursive Attention Feature Fusion Network for Image Super-Resolution Reconstruction Algorithm

Image supper resolution by dilated dense progressive network

Progressive Splitting and Upscaling Structure for Super-Resolution

Multi-Scale Deep Neural Networks for Real Image Super-Resolution

Deep SR-HDR: Joint Learning of Super-Resolution and High Dynamic Range Imaging for Dynamic Scenes

Feature Fusion Super Resolution Network with Gradient Guidance

Image Super-Resolution Reconstruction Model Based on Multi-Feature Fusion