Abstract:The exponential performance growth guaranteed by Moore’s law has started to taper in recent years. At the same time, emerging applications like image processing demand heavy computational performance. These factors inevitably lead to the emergence of domain-specific accelerators (DSA) to fill the performance void left by conventional architectures. FPGAs are rapidly evolving towards becoming an alternative to custom ASICs for designing DSAs because of their low power consumption and a higher degree of parallelism. DSA design on FPGAs requires careful calibration of the FPGA compute and memory resources towards achieving optimal throughput. Hardware Descriptive Languages (HDL) like Verilog have been traditionally used to design FPGA hardware. HDLs are not geared towards any domain, and the user has to put in much effort to describe the hardware at the register transfer level. Domain Specific Languages (DSLs) and compilers have been recently used to weave together handwritten HDLs templates targeting a specific domain. Recent efforts have designed DSAs with image-processing DSLs targeting FPGAs. Image computations in the DSL are lowered to pre-existing templates or lower-level languages like HLS-C. This approach requires expensive FPGA re-flashing for every new workload. In contrast to this fixed-function hardware approach, overlays are gaining traction. Overlays are DSAs resembling a processor, which is synthesized and flashed on the FPGA once but is flexible enough to process a broad class of computations through soft reconfiguration. Less work has been reported in the context of image processing overlays. Image processing algorithms vary in size and shape, ranging from simple blurring operations to complex pyramid systems. The primary challenge in designing an image-processing overlay is maintaining flexibility in mapping different algorithms. This paper proposes a DSL-based overlay accelerator called FlowPix for image processing applications. The DSL programs are expressed as pipelines, with each stage representing a computational step in the overall algorithm. We implement 15 image-processing benchmarks using FlowPix on a Virtex-7-690t FPGA. The benchmarks range from simple blur operations to complex pipelines like Lucas-Kande optical flow. We compare FlowPix against existing DSL-to-FPGA frameworks like Hetero-Halide and Vitis Vision library that generate fixed-function hardware. On most benchmarks, we see up to 25% degradation in latency with approximately a 1.7x to 2x increase in the FPGA LUT consumption. Our ability to execute any benchmark without incurring the high costs of hardware synthesis, place-and-route, and FPGA re-flashing justifies the slight performance loss and increased resource consumption that we experience. FlowPix achieves an average frame rate of 170 FPS on HD frames of 1920x1080 pixels in the implemented benchmarks.

A FPGA Implementation of Farneback Optical Flow by High-Level Synthesis

An FPGA-Optimized Architecture of Real-time Farneback Optical Flow

Real-time FPGA Implementation of an Optic Flow Algorithm Suitable for Background Oriented Schlieren Technique

FPGA-based hardware implementation of new edge-directed interpolation algorithm

FPGA Implementation of Feature Detection Algorithm Based on High Level Synthesis

High speed video signal acquisition and processing system based on FPGA technique

Real-Time Efficient FPGA Implementation of the Multi-Scale Lucas-Kanade and Horn-Schunck Optical Flow Algorithms for a 4K Video Stream

FPGA implementation of full HD real-time depth estimation

FPGA-Based Feature Extraction and Tracking Accelerator for Real-Time Visual SLAM

High-performance architecture for real-time high-definition short-wave infrared streaming video processing and its field programmable gate array prototype

NeuFlow v2: High-Efficiency Optical Flow Estimation on Edge Devices

FlowPix: Accelerating Image Processing Pipelines on an FPGA Overlay using a Domain Specific Compiler

Pflow: An end-to-end heterogeneous acceleration framework for CNN inference on FPGAs

fpgaHART: A toolflow for throughput-oriented acceleration of 3D CNNs for HAR onto FPGAs

Parallel Hough Transform-based straight line detection and its FPGA implementation in embedded vision

The hardware implementation of adaptive non-linear sampling algorithm

Dataflow Optimized Reconfigurable Acceleration for FEM-based CFD Simulations

NeuFlow: Real-time, High-accuracy Optical Flow Estimation on Robots Using Edge Devices

High-level synthesis: productivity, performance, and software constraints

FlowNAS: Neural Architecture Search for Optical Flow Estimation

Revisiting FPGA Acceleration of Molecular Dynamics Simulation with Dynamic Data Flow Behavior in High-Level Synthesis