Abstract:Super-resolution (SR) techniques aim to restore a high-resolution (HR) image from low-resolution (LR) images, which are often used to assist the enhancement of image/video quality under the rapid development of HR and high-frame-rate media. Recently, neural network (NN)-based methods perform much better image reconstruction quality than classical approaches. However, the unacceptable computation complexity as well as the huge memory footprints of NNs limit the throughputs and scalability of these SR systems. In this work, we analyze several key issues in the design of NN-based SR systems first. Then, we propose a three-level systematic optimization methodology for SR systems to reduce computation overhead and keep image quality. At the algorithm level, we introduce image blocking to SR tasks and develop a block-wise SR algorithm based on the hybrid of NN and interpolation with a consistent image block evaluation metric. The configurable hybrid parameters help the SR algorithm to achieve a flexible tradeoff between the computation overhead and image quality. At the operator level, we focus on the transpose convolution operators commonly used for upsampling in SR NNs. We propose an efficient Winograd-based transposed convolution acceleration method. Through the efficient subconvolutions conversion and the Winograd specialization, this methods enables unified Winograd transformations and simplified data access patterns. At the data level, we propose a novel quantization method for Winograd-aware SR NNs to get better-quantized accuracy. Comprehensive evaluations demonstrate the effectiveness of these optimizations. Our SR system reduces a large number of multiplications with great scalability and supports 4K@120 fps and 8K@30 fps outputs with acceptable image quality degradation.

TileSR: Accelerate On-Device Super-Resolution with Parallel Offloading in Tile Granularity

NoUCSR: Efficient Super-Resolution Network Without Upsampling Convolution.

SplitSR: An End-to-End Approach to Super-Resolution on Mobile Devices

Super-Resolution by Predicting Offsets: An Ultra-Efficient Super-Resolution Network for Rasterized Images.

TBSR: Tile-Based 360° Video Streaming with Super-Resolution on Commodity Mobile Devices

Single Remote Sensing Image Super-Resolution Via a Generative Adversarial Network with Stratified Dense Sampling and Chain Training

NAWQ-SR: A Hybrid-Precision NPU Engine for Efficient On-Device Super-Resolution

Edge-SD-SR: Low Latency and Parameter Efficient On-device Super-Resolution with Stable Diffusion via Bidirectional Conditioning

Hardware-friendly Scalable Image Super Resolution with Progressively Structured Sparsity

Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolution

Mob-FGSR: Frame Generation and Super Resolution for Mobile Real-Time Rendering

Texture-Enhanced Framework by Differential Filter-Based Re-parameterization for Super-Resolution on PC/Mobile

Anchor-based Plain Net for Mobile Image Super-Resolution

ESSR: An 8K@30FPS Super-Resolution Accelerator With Edge Selective Network

A 28-nm Computing-in-Memory-Based Super-Resolution Accelerator Incorporating Macro-Level Pipeline and Texture/Algebraic Sparsity

Efficient Super-Resolution System with Block-Wise Hybridization and Quantized Winograd on FPGA

Real-Time Super-Resolution for Real-World Images on Mobile Devices

Reparameterizable Multibranch Bottleneck Network for Lightweight Image Super-Resolution

An Improved Image Super Resolution and Its Parallel Implementation Based on CUDA.

AsConvSR: Fast and Lightweight Super-Resolution Network with Assembled Convolutions

From Coarse to Fine: Hierarchical Pixel Integration for Lightweight Image Super-resolution