Abstract:Video super-resolution aims to recover the high-resolution (HR) contents from the low-resolution (LR) observations relying on compositing the spatial-temporal information in the LR frames. It is crucial to model the spatial-temporal information jointly since the video sequences are three-dimensional spatial-temporal signals. Compared with explicitly estimating motions between the 2D frames, 3D convolutional neural networks (CNNs) have been shown its efficiency and effectiveness for video super-resolution (SR), as a natural way of spatial-temporal data modelling. Though promising, the performance of 3D CNNs is still far from satisfactory. The high computational and memory requirements limit the development of more advanced designs to extract and fuse the information from a larger spatial and temporal scale. We thus propose a Mixed Spatial-Temporal Convolution (MSTC) block that simultaneously extracts the spatial information and the supplemented temporal dependency among frames by jointly applying 2D and 3D convolution. To further fuse the learned features corresponding to different frames, we propose a novel similarity-based selective features strategy, unlike precious methods directly stacking the learned features. Additionally, an attention-based motion compensation module is applied to alleviate the influence of misalignment between frames. Experiments on three widely used benchmark datasets and real-world dataset show that, relying on superior feature extraction and fusion ability, the proposed network can outperform previous state-of-the-art methods, especially for recovering the confusing details.

Deformable 3D Convolution for Video Super-Resolution

Spatio-Temporal Deformable Convolution for Compressed Video Quality Enhancement

3D Deformable Kernels for Video Super-Resolution

Video super-resolution with phase-aided deformable alignment network

Fast Spatio-Temporal Residual Network For Video Super-Resolution

Iterative Back Projection Network Based on Deformable 3D Convolution

Video super-resolution with 3D adaptive normalized convolution

An Efficient Accelerator of Deformable 3D Convolutional Network for Video Super-Resolution.

Video super-resolution via mixed spatial-temporal convolution and selective fusion

Structure-preserving video super-resolution using three-dimensional convolutional neural networks

3D Video Super-Resolution Using Fully Convolutional Neural Networks

Video Super-Resolution Via a Spatio-Temporal Alignment Network.

Deformable Kernel Convolutional Network for Video Extreme Super-Resolution

Grouped Spatio-Temporal Alignment Network for Video Super-Resolution

Video super-resolution via dense non-local spatial-temporal convolutional network

Temporal Modulation Network for Controllable Space-Time Video Super-Resolution

AsConvSR: Fast and Lightweight Super-Resolution Network with Assembled Convolutions

Efficient Spatio-Temporal Network with Gated Fusion for Video Super-Resolution

Deep Plug-and-Play Video Super-Resolution.

Enhanced Video Super-Resolution Network Towards Compressed Data

3DAttGAN: A 3D Attention-based Generative Adversarial Network for Joint Space-Time Video Super-Resolution