Abstract:In this paper, we propose a hardware-efficient residual recurrent neural network for real-time video super-resolution (VSR) based on field programmable gate array (FPGA). Although recent learning-based VSR methods have achieved remarkable performance, the large computational complexity prohibits the deployment of the sophisticated VSR models on FPGA for real-time applications. Limited by the hardware resources, state-of-the-art FPGA-based VSR methods perform single-image super-resolution over the video sequence and suffer from temporal inconsistency. In order to exploit the inter-frame temporal correlation for real-time VSR on low-complexity hardware, we introduce a hardware-efficient recurrent neural network ERVSR. Specially, the proposed ERVSR leverages the input frame and the temporal information entailed in the hidden state to reconstruct the high-resolution counterpart. To reduce the network parameters, the low-resolution input branch and the hidden state branch are convolved individually and a channel modulation coefficient is proposed to explicitly guide the network to allocate the amount of output feature channels to each branch. Additionally, in order to reduce the memory consumption, we perform a dedicated lightweight compression of the hidden state by introducing a statistical normalization scheme followed by a fixed-point quantization. Besides, we adopt group convolution and depthwise separable convolution to further compact the network. We evaluated the proposed ERVSR on multiple public datasets from different aspects. Experimental results demonstrate that ERVSR performs better than the existing state-of-the-art FPGA-based VSR methods in both image quality and data throughput.

FPGA-Based Real-Time Super-Resolution System for Ultra High Definition Videos

Live Video Analytics with FPGA-based Smart Cameras

High speed video signal acquisition and processing system based on FPGA technique

Winograd-Based Real-Time Super-Resolution System on FPGA

Design and Implementation of Visual Processing System Based on FPGA

A novel high dynamic range microscopic video system based on REC.2020

A Real-Time Full HD 2D-to-3d Video Conversion System Based on FPGA

FPGA implementation of full HD real-time depth estimation

Full Hd Real-Time Depth Estimation Algorithm and Hardware Implementation

FPGA Implementation of Non-Parametric Stereo Matching Algorithm

Real-time 3D Video System Based on FPGA

A High-Performance Accelerator for Real-Time Super-Resolution on Edge FPGAs

Full High-Definition Real-Time Depth Estimation for Three-Dimensional Video System

Decoded Ultra-High Definition Video Display System

An FPGA-Based Residual Recurrent Neural Network for Real-Time Video Super-Resolution

A Video Coding System with Spatial-Temporal Down-/Up-Sampling and Super-Resolution Reconstruction

A real-time multi-view interlacing architecture for auto-stereoscopic 3DTV display based on FPGA

Real-time video processing model for stereoscopic display system based on FPGA

High-performance architecture for real-time high-definition short-wave infrared streaming video processing and its field programmable gate array prototype

FPGA Implementation of Feature Detection Algorithm Based on High Level Synthesis

A wide-spectrum and high-speed CMOS system based on FPGA