Space-time video super-resolution via multi-scale feature interpolation and temporal feature fusion
Caisong Yang,Guangqian Kong,Xun Duan,Huiyun Long,Jian Zhao
DOI: https://doi.org/10.1007/s11760-024-03469-7
IF: 1.583
2024-09-15
Signal Image and Video Processing
Abstract:The goal of Space-Time Video Super-Resolution (STVSR) is to simultaneously increase the spatial resolution and frame rate of low-resolution, low-frame-rate video. In response to the problem that the STVSR method does not fully consider the spatio-temporal correlation between successive video frames, which makes the video frame reconstruction results unsatisfactory, and the problem that the inference speed of large models is slow. This paper proposes a STVSR method based on Multi-Scale Feature Interpolation and Temporal Feature Fusion (MSITF). First, feature interpolation is performed in the low-resolution feature space to obtain the features corresponding to the missing frames. The feature is then enhanced using deformable convolution with the aim of obtaining a more accurate feature of the missing frames. Finally, the temporal alignment and global context learning of sequence frame features are performed by a temporal feature fusion module to fully extract and utilize the useful spatio-temporal information in adjacent frames, resulting in better quality of the reconstructed video frames. Extensive experiments on the benchmark datasets Vid4 and Vimeo-90k show that the proposed method achieves better qualitative and quantitative performance, with PSNR and SSIM on the Vid4 dataset improving by 0.8% and 1.9%, respectively, over the state-of-the-art two-stage method AdaCof+TTVSR, and MSITF improved by 1.2% and 2.5%, respectively, compared to single-stage method RSTT. The number of parameters decreased by 80.4% and 8.2% compared to the AdaCof+TTVSR and RSTT, respectively.We release our code at https://github.com/carpenterChina/MSITF.
engineering, electrical & electronic,imaging science & photographic technology