Abstract:Modern consumer cameras usually employ the rolling shutter (RS) mechanism, where images are captured by scanning scenes row-by-row, yielding RS distortions for dynamic scenes. To correct RS distortions, existing methods adopt a fully supervised learning manner, where high framerate global shutter (GS) images should be collected as ground-truth supervision. In this paper, we propose a Self-supervised learning framework for Dual reversed RS distortions Correction (SelfDRSC), where a DRSC network can be learned to generate a high framerate GS video only based on dual RS images with reversed distortions. In particular, a bidirectional distortion warping module is proposed for reconstructing dual reversed RS images, and then a self-supervised loss can be deployed to train DRSC network by enhancing the cycle consistency between input and reconstructed dual reversed RS images. Besides start and end RS scanning time, GS images at arbitrary intermediate scanning time can also be supervised in SelfDRSC, thus enabling the learned DRSC network to generate a high framerate GS video. Moreover, a simple yet effective self-distillation strategy is introduced in self-supervised loss for mitigating boundary artifacts in generated GS images. On synthetic dataset, SelfDRSC achieves better or comparable quantitative metrics in comparison to state-of-the-art methods trained in the full supervision manner. On real-world RS cases, our SelfDRSC can produce high framerate GS videos with finer correction textures and better temporary consistency. The source code and trained models are made publicly available at <a class="link-external link-https" href="https://github.com/shangwei5/SelfDRSC" rel="external noopener nofollow">this https URL</a>. We also provide an implementation in HUAWEI Mindspore at <a class="link-external link-https" href="https://github.com/Hunter-Will/SelfDRSC-mindspore" rel="external noopener nofollow">this https URL</a>.

Learning Bilateral Cost Volume for Rolling Shutter Temporal Super-Resolution

Self-Supervised Learning for Rolling Shutter Temporal Super-Resolution

Context-Aware Video Reconstruction for Rolling Shutter Cameras

Single Remote Sensing Image Super-Resolution Via a Generative Adversarial Network with Stratified Dense Sampling and Chain Training

Two-Stage Spatial-Frequency Joint Learning for Large-Factor Remote Sensing Image Super-Resolution

Video Super-Resolution Via a Spatio-Temporal Alignment Network.

You Only Align Once: Bidirectional Interaction for Spatial-Temporal Video Super-Resolution

Deep Plug-and-Play Video Super-Resolution.

Enhanced Video Super-Resolution Network Towards Compressed Data

Video Super-Resolution Reconstruction Based on Deep Convolutional Neural Network and Spatio-Temporal Similarity

Self-supervised Learning to Bring Dual Reversed Rolling Shutter Images Alive

SelfDRSC++: Self-Supervised Learning for Dual Reversed Rolling Shutter Correction

Video super-resolution via event-driven temporal alignment

Optical Flow Reusing for High-Efficiency Space-Time Video Super Resolution

Enhanced Bidirectional Propagation Network for Video Super-Resolution

Learning Spatial-Temporal Implicit Neural Representations for Event-Guided Video Super-Resolution

Revisiting Temporal Modeling for Video Super-resolution.

Spatial-Temporal Space Hand-in-Hand: Spatial-Temporal Video Super-Resolution Via Cycle-Projected Mutual Learning

Look Back and Forth: Video Super-Resolution with Explicit Temporal Difference Modeling

Spatiotemporal Super-Resolution Reconstruction Based on Robust Optical Flow and Zernike Moment for Video Sequences

RS-NeRF: Neural Radiance Fields from Rolling Shutter Images