Abstract:INFerence-as-a-Service (INFaaS) has become a primary workload in the cloud. However, existing FPGA-based Deep Neural Network (DNN) accelerators are mainly optimized for the fastest speed of a single task, while the multi-tenancy of INFaaS has not been explored yet. As the demand for INFaaS keeps growing, simply increasing the number of FPGA-based DNN accelerators is not cost-effective, while merely sharing these single-task optimized DNN accelerators in a time-division multiplexing way could lead to poor isolation and high-performance loss for INFaaS. On the other hand, current cloud-based DNN accelerators have excessive compilation overhead, especially when scaling out to multi-FPGA systems for multi-tenant sharing, leading to unacceptable compilation costs for both offline deployment and online reconfiguration. Therefore, it is far from providing efficient and flexible FPGA virtualization for public and private cloud scenarios. Aiming to solve these problems, we propose a unified virtualization framework for general-purpose deep neural networks in the cloud, enabling multi-tenant sharing for both the Convolution Neural Network (CNN), and the Recurrent Neural Network (RNN) accelerators on a single FPGA. The isolation is enabled by introducing a two-level instruction dispatch module and a multi-core based hardware resources pool. Such designs provide isolated and runtime-programmable hardware resources, which further leads to performance isolation for multi-tenant sharing. On the other hand, to overcome the heavy re-compilation overheads, a tiling-based instruction frame package design and a two-stage static-dynamic compilation, are proposed. Only the lightweight runtime information is re-compiled with ∼1 ms overhead, thus guaranteeing the private cloud’s performance. Finally, the extensive experimental results show that the proposed virtualized solutions achieve up to 3.12× and 6.18× higher throughput in the private cloud compared with the static CNN and RNN baseline designs, respectively.

FiC-RNN: A Multi-FPGA Acceleration Framework for Deep Recurrent Neural Networks

An LSTM Acceleration Engine for FPGAs Based on Caffe Framework

Cloud Server Oriented FPGA Accelerator for Long Short-Term Memory Recurrent Neural Networks

An Fpga-Based Lstm Acceleration Engine For Deep Learning Frameworks

A Cloud Server Oriented FPGA Accelerator for LSTM Recurrent Neural Network

FPGA-based Accelerator for Long Short-Term Memory Recurrent Neural Networks

A Unified FPGA Virtualization Framework for General-Purpose Deep Neural Networks in the Cloud

FP-DNN: an Automated Framework for Mapping Deep Neural Networks Onto FPGAs with RTL-HLS Hybrid Templates

Implementation and Optimization of the Accelerator Based on FPGA Hardware for LSTM Network

An Efficient Reconfigurable Framework for General Purpose CNN-RNN Models on FPGAs

Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inference

Toward Full-Stack Acceleration of Deep Convolutional Neural Networks on FPGAs

A multistage dataflow implementation of a Deep Convolutional Neural Network based on FPGA for high-speed object recognition

ConvCloud: an Adaptive Convolutional Neural Network Accelerator on Cloud FPGAs.

FPDeep: Scalable Acceleration of CNN Training on Deeply-Pipelined FPGA Clusters

E-RNN: Design Optimization for Efficient Recurrent Neural Networks in FPGAs

FPGA Acceleration of Recurrent Neural Network Based Language Model

Pflow: An end-to-end heterogeneous acceleration framework for CNN inference on FPGAs

XFER: A Novel Design to Achieve Super-Linear Performance on Multiple FPGAs for Real-Time AI.

A Deep Residual Networks Accelerator on FPGA

An OpenCL-Based FPGA Accelerator for Faster R-CNN