Multi-Reference Image Super-Resolution: A Posterior Fusion Approach

Ke Zhao,Haining Tan,Tsz Fung Yau

DOI: https://doi.org/10.48550/arXiv.2212.09988

2022-12-20

Abstract:Reference-based Super-resolution (RefSR) approaches have recently been proposed to overcome the ill-posed problem of image super-resolution by providing additional information from a high-resolution image. Multi-reference super-resolution extends this approach by allowing more information to be incorporated. This paper proposes a 2-step-weighting posterior fusion approach to combine the outputs of RefSR models with multiple references. Extensive experiments on the CUFED5 dataset demonstrate that the proposed methods can be applied to various state-of-the-art RefSR models to get a consistent improvement in image quality.

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the problem of insufficient information in single - image super - resolution (SISR). Specifically, traditional SISR methods face a serious under - determined problem when reconstructing high - resolution (HR) images from low - resolution (LR) images, that is, there are countless possible solutions that satisfy the given measurement conditions. To overcome this challenge, the reference - image super - resolution (RefSR) method has been proposed, which provides additional information by introducing an external high - resolution reference image to improve the effect of super - resolution.

However, most of the existing RefSR models can only handle a single reference image, which limits the amount of supplementary information that can be utilized. To solve this problem, this paper proposes a multi - reference - image super - resolution (MRefSR) method and introduces a **posterior fusion strategy**, aiming to combine the information provided by multiple reference images and further improve the quality of super - resolution images.

### Specific problem description

1. **Limitations of single - reference images**: Most of the existing RefSR models can only use one reference image and cannot fully utilize the rich information provided by multiple reference images.
2. **Fusion challenges of multi - reference images**: How to effectively fuse the information from multiple reference images to generate high - quality super - resolution images is an urgent problem to be solved.
3. **Balance between computational efficiency and effect**: When implementing multi - reference - image fusion, it is necessary to ensure both computational efficiency and a significant improvement in the quality of the finally generated image.

### Solutions proposed in the paper

The paper proposes a two - step - weighting posterior fusion approach, which can fuse the outputs of multiple RefSR models to obtain higher - quality super - resolution images. The specific steps are as follows:

1. **Image alignment and texture extraction module**: First, use the existing single - reference - image super - resolution model to process the input low - resolution image and multiple high - resolution reference images to generate multiple super - resolution output images.
2. **Image fusion module**:
- **Adaptive weight masking**: Calculate an adaptive weight mask \( W_i \) for each super - resolution output image. This mask assigns different weights according to the reconstruction quality of the pixel area.
- **Global reference - quality - based weighted averaging**: Perform weighted averaging on multiple super - resolution output images based on the global reference - quality - based weight \( w_i \), and finally generate the fused super - resolution image.

Through this method, the paper not only improves the quality of super - resolution images but also shows the potential of its method in various application scenarios, such as video - game - graphics rendering.

### Summary

The core objective of this paper is to solve the problem of insufficient information in single - image super - resolution by introducing multi - reference images and adopting an effective fusion strategy, thereby significantly improving the quality of super - resolution images.