Deformable Convolution Alignment and Dynamic Scale-Aware Network for Continuous-Scale Satellite Video Super-Resolution
Ning Ni,Libao Zhang
DOI: https://doi.org/10.1109/tgrs.2024.3366550
IF: 8.2
2024-02-27
IEEE Transactions on Geoscience and Remote Sensing
Abstract:Recently, due to higher requirements for satellite video resolution, video super-resolution (VSR) has been extensively studied. However, the following problems have not been effectively resolved: 1) previous satellite VSR methods cannot achieve continuous-scale (integer and noninteger scale) VSR with a single model; 2) satellite video has complex ground and weak textures, which increases the difficulty of capturing motion information. In addition, existing methods adopt a unified alignment path, which leads to a drop in feature alignment accuracy; and 3) during feature fusion, previous methods ignore the correlation of spatiotemporal information in satellite video and cannot make full use of the spatiotemporal information. To address the above problems, in this article, we propose a novel network for continuous-scale satellite VSR (CSVSR). Specifically, first, for effective motion capture and accurate feature alignment, we design a residual-guided and time-aware dynamic routing alignment module, which can use feature residuals to lock motion areas and then dynamically select the corresponding alignment path based on the temporal distance. Second, we proposed a nonlocal mask-based feature fusion module (NMFFM) to exploit the correlation of the spatiotemporal features and complete effective spatiotemporal feature fusion. Third, to make our network adapt to multitask learning, we develop a scale-aware convolutional (SA-Conv) layer, which lets our network dynamically extract scale-adaptive features according to the input scale factors. Finally, we propose a continuous-scale upsampling module with a global feature implicit function (GFIF), which can achieve continuous-scale mapping from features to pixel values. In addition, we carefully design a novel training strategy to optimize our network. Comprehensive experiments verify that the proposed CSVSR has superior reconstruction performance on continuous-scale factors. The code will be available at https://github.com/chongningni/CSVSR.
imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geochemistry & geophysics