Abstract:Traditional Convolutional Neural Network (ConvNet, CNN)-based image super-resolution (SR) methods have lower computation costs, making them more friendly for real-world scenarios. However, they suffer from lower performance. On the contrary, Vision Transformer (ViT)-based SR methods have achieved impressive performance recently, but these methods often suffer from high computation costs and model storage overhead, making them hard to meet the requirements in practical application scenarios. In practical scenarios, an SR model should reconstruct an image with high quality and fast inference. To handle this issue, we propose a novel CNN-based Efficient Residual ConvNet enhanced with structural Re-parameterization (RepECN) for a better trade-off between performance and efficiency. A stage-to-block hierarchical architecture design paradigm inspired by ViT is utilized to keep the state-of-the-art performance, while the efficiency is ensured by abandoning the time-consuming Multi-Head Self-Attention (MHSA) and by re-designing the block-level modules based on CNN. Specifically, RepECN consists of three structural modules: a shallow feature extraction module, a deep feature extraction, and an image reconstruction module. The deep feature extraction module comprises multiple ConvNet Stages (CNS), each containing 6 Re-Parameterization ConvNet Blocks (RepCNB), a head layer, and a residual connection. The RepCNB utilizes larger kernel convolutions rather than MHSA to enhance the capability of learning long-range dependence. In the image reconstruction module, an upsampling module consisting of nearest-neighbor interpolation and pixel attention is deployed to reduce parameters and maintain reconstruction performance, while bicubic interpolation on another branch allows the backbone network to focus on learning high-frequency information. The extensive experimental results on multiple public benchmarks show that our RepECN can achieve 2.5∼5× faster inference than the state-of-the-art ViT-based SR model with better or competitive super-resolving performance, indicating that our RepECN can reconstruct high-quality images with fast inference.

EViTIB: Efficient Vision Transformer Via Inductive Bias Exploration for Image Super-Resolution

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Super Vision Transformer

Convolutional Embedding Makes Hierarchical Vision Transformer Stronger

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

Deep Transformers Thirst for Comprehensive-Frequency Data

CRViT: Vision transformer advanced by causality and inductive bias for image recognition

RepECN: Making ConvNets Better Again for Efficient Image Super-Resolution

HIRI-ViT: Scaling Vision Transformer with High Resolution Inputs

LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition

AdaViT: Adaptive Vision Transformers for Efficient Image Recognition

DRCT: Saving Image Super-resolution away from Information Bottleneck

Improving Vision Transformers by Revisiting High-Frequency Components

HiT-SR: Hierarchical Transformer for Efficient Image Super-Resolution

CF-ViT: A General Coarse-to-Fine Method for Vision Transformer

Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets

Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets

EViT: An Eagle Vision Transformer with Bi-Fovea Self-Attention

Hierarchical Separable Video Transformer for Snapshot Compressive Imaging

DctViT: Discrete Cosine Transform Meet Vision Transformers

Effective Diffusion Transformer Architecture for Image Super-Resolution