Image Super-Resolution with Implicit Texture Pattern Modulation

Shuai Hao,Jialin Yang,Xu Jia,You He,Huchuan Lu

DOI: https://doi.org/10.1109/ICME55011.2023.00217

2023-01-01

Abstract:Image super-resolution is one of the classical low-level vision tasks with the purpose of restoring a high-resolution image with fine details. Being aware of texture patterns with an image would benefit super-resolution performance a lot. However, it would be difficult to predict texture patterns for each region because of lack of annotations with different kinds of categories. In this work, we propose to implicitly model texture information with each region and take that as prior to promote super-resolution performance. In order to fully explore texture patterns, a hybrid model of convolutional neural networks and transformers is proposed. It is able to take advantage of both local and long range dependencies within an image to super-resolve an image. Specifically, a set of transformers are employed to model self-similarity among pixels within an image and to make dense texture patterns prediction at each pixel. The computed texture pattern representation then works as condition to modulate convolution-based residual blocks. In this way texture patterns could be integrated into the CNNs to obtain powerful features for image super-resolution. Extensive experiments on several benchmark datasets demonstrate its favorable performance against state-of-the-art methods and show its potential as a generic design for hybrid of transformers and CNNs.