Abstract:Super-resolution (SR) techniques aim to restore a high-resolution (HR) image from low-resolution (LR) images, which are often used to assist the enhancement of image/video quality under the rapid development of HR and high-frame-rate media. Recently, neural network (NN)-based methods perform much better image reconstruction quality than classical approaches. However, the unacceptable computation complexity as well as the huge memory footprints of NNs limit the throughputs and scalability of these SR systems. In this work, we analyze several key issues in the design of NN-based SR systems first. Then, we propose a three-level systematic optimization methodology for SR systems to reduce computation overhead and keep image quality. At the algorithm level, we introduce image blocking to SR tasks and develop a block-wise SR algorithm based on the hybrid of NN and interpolation with a consistent image block evaluation metric. The configurable hybrid parameters help the SR algorithm to achieve a flexible tradeoff between the computation overhead and image quality. At the operator level, we focus on the transpose convolution operators commonly used for upsampling in SR NNs. We propose an efficient Winograd-based transposed convolution acceleration method. Through the efficient subconvolutions conversion and the Winograd specialization, this methods enables unified Winograd transformations and simplified data access patterns. At the data level, we propose a novel quantization method for Winograd-aware SR NNs to get better-quantized accuracy. Comprehensive evaluations demonstrate the effectiveness of these optimizations. Our SR system reduces a large number of multiplications with great scalability and supports 4K@120 fps and 8K@30 fps outputs with acceptable image quality degradation.

Winograd-Based Real-Time Super-Resolution System on FPGA

Design and Implementation of Visual Processing System Based on FPGA

FPGA-Based Real-Time Super-Resolution System for Ultra High Definition Videos

Efficient Super-Resolution System with Block-Wise Hybridization and Quantized Winograd on FPGA

High speed video signal acquisition and processing system based on FPGA technique

A Real-Time Full HD 2D-to-3d Video Conversion System Based on FPGA

FPGA implementation of full HD real-time depth estimation

Improved real-time correlation-based FPGA stereo vision system

FPGA Implementation of Non-Parametric Stereo Matching Algorithm

Full Hd Real-Time Depth Estimation Algorithm and Hardware Implementation

A real-time multi-view interlacing architecture for auto-stereoscopic 3DTV display based on FPGA

An FPGA-Based Residual Recurrent Neural Network for Real-Time Video Super-Resolution

Real-time video processing model for stereoscopic display system based on FPGA

A High-Performance Accelerator for Real-Time Super-Resolution on Edge FPGAs

Real-Time High-Quality Stereo Vision System in FPGA

Neural Super-Resolution in Real-Time Rendering Using Auxiliary Feature Enhancement

A High-efficiency FPGA-based Accelerator for Convolutional Neural Networks using Winograd Algorithm

Real-time 3D Video System Based on FPGA

FPGA based memory efficient high resolution stereo vision system for video tolling

FPGA Implementation of Feature Detection Algorithm Based on High Level Synthesis

FuseSR: Super Resolution for Real-time Rendering through Efficient Multi-resolution Fusion