Learning Dual-Level Deformable Implicit Representation for Real-World Scale Arbitrary Super-Resolution

Zhiheng Li,Muheng Li,Jixuan Fan,Lei Chen,Yansong Tang,Jie Zhou,Jiwen Lu

2024-03-16

Abstract:Scale arbitrary super-resolution based on implicit image function gains increasing popularity since it can better represent the visual world in a continuous manner. However, existing scale arbitrary works are trained and evaluated on simulated datasets, where low-resolution images are generated from their ground truths by the simplest bicubic downsampling. These models exhibit limited generalization to real-world scenarios due to the greater complexity of real-world degradations. To address this issue, we build a RealArbiSR dataset, a new real-world super-resolution benchmark with both integer and non-integer scaling factors for the training and evaluation of real-world scale arbitrary super-resolution. Moreover, we propose a Dual-level Deformable Implicit Representation (DDIR) to solve real-world scale arbitrary super-resolution. Specifically, we design the appearance embedding and deformation field to handle both image-level and pixel-level deformations caused by real-world degradations. The appearance embedding models the characteristics of low-resolution inputs to deal with photometric variations at different scales, and the pixel-based deformation field learns RGB differences which result from the deviations between the real-world and simulated degradations at arbitrary coordinates. Extensive experiments show our trained model achieves state-of-the-art performance on the RealArbiSR and RealSR benchmarks for real-world scale arbitrary super-resolution. Our dataset as well as source code will be publicly available.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper focuses on the problem of Real-World Scale Arbitrary Super-Resolution. Existing methods usually train and evaluate on simulated datasets, which generate low-resolution images using simple bilinear downsampling. However, this approach has limited generalization ability for the complex degradation situations in the real world.

To address this issue, the paper proposes a new dataset called RealArbiSR, which serves as a benchmark for training and evaluating real-world scale arbitrary super-resolution. The dataset includes both integer and non-integer scaling factors. Additionally, they propose a method called "Dual-Level Deformable Implicit Representation" (DDIR) that handles image-level and pixel-level deformations caused by real-world degradations by designing appearance embeddings and deformation fields.

Experiments show that the DDIR model achieves the best performance in real-world scale arbitrary super-resolution on the RealArbiSR and RealSR benchmarks. The paper also compares it with current CNN-based and implicit neural representation-based methods and points out their limitations in dealing with real-world scenarios.

In summary, the main contributions of the paper are:

1. Creating the RealArbiSR dataset, which is the first real-world super-resolution benchmark with both integer and non-integer scaling factors.
2. Proposing a Dual-Level Deformable Implicit Representation (DDIR) that learns and handles image deformations caused by complex real-world degradations.
3. Experimental results demonstrate that the DDIR model achieves state-of-the-art performance in real-world scale arbitrary super-resolution tasks.