Multi-scale Non-local Bidirectional Fusion for Video Super-Resolution

Quan Zhou,Qiong Liu,Fen Chen,Sheng Wang,Zongju Peng

DOI: https://doi.org/10.1007/978-3-031-46317-4_15

2023-01-01

Abstract:Long-range dependency is one of the important inscriptions in sequence modeling. For video data, the commonly used convolutional and recurrent operations are a kind of “local coding” for variable-length sequences, which can only capture the local neighborhood information. We introduce the idea of non-local mean to compensate for the shortcomings of repeated convolutional operations, while most of the previous non-local methods used for video super-resolution only focus on positional information or fail to capture temporal information directly. In this study, we propose a non-local bidirectional fusion network (NLBF) for the video super-resolution (VSR) task. This non-local network decouples multidimensional information to reduce computational memory consumption, at the same time capturing long-range dependencies within the temporal-spatial-channel dimension as much as possible. In the multi-scale local and non-local hybrid framework, we further design the bidirectional spatial-temporal fusion module to balance the information obtained from other frames while achieving feature refinement. Experimental results on benchmark datasets show that the proposed NLBF is able to achieve state-of-the-art performance in the VSR task.