Frame-Rate-Aware Aggregation for Efficient Video Super-Resolution

Takashi Isobe,Fang Zhu,Shengjin Wang

DOI: https://doi.org/10.1109/icassp39728.2021.9414334

2021-01-01

Abstract:Video super-resolution, which aims at producing a high-resolution video from its corresponding low-resolution version, recently draws increasing attention. In contrast to the previous works that perform explicit motion estimation and compensation, we propose a novel deep neural network which performs implicit motion estimation with frame-rate-based temporal aggregation. Specifically, the input frames are first aggregated by a frame-rate-aware 3D convolution layer, where neighboring frames are integrated with the reference frame according to the corresponding frame rate. Then, the aggregated features are fed into several branches for further aggregation. Different branches correspond to a kind of motion rate, which provides complementary information to recover missing details in the reference frame. Extensive experiments demonstrate that our method is able to handle various motion types and achieves state-of-the-art performance on several benchmarks. In addition, our model is light-weight and requires an extremely less computational load than other state-of-the-art methods.