Abstract:With the recent advancement of multilayer convolutional neural networks (CNNs) and fully connected networks (FCNs), deep learning has achieved amazing success in many areas, especially in visual content understanding and classification. To improve the performance and energy efficiency of the computation-demanding CNN, the FPGA-based acceleration emerges as one of the most attractive alternatives. In this paper, we design and implement Caffeine, a hardware/software co-designed library to efficiently accelerate the entire CNN and FCN on FPGAs. First, we propose a uniformed convolutional matrix-multiplication representation for both computation-bound convolutional layers and communication-bound FCN layers. Based on this representation, we optimize the accelerator micro-architecture and maximize the underlying FPGA computing and bandwidth resource utilization based on a revised roofline model. Moreover, we design an automation flow to directly compile high-level network definitions to the final FPGA accelerator. As a case study, we integrate Caffeine into the industry-standard software deep learning framework Caffe. We evaluate Caffeine and its integration with Caffe by implementing VGG16 and AlexNet networks on multiple FPGA platforms. Caffeine achieves a peak performance of 1460 giga fixed point operations per second on a medium-sized Xilinx KU060 FPGA board; to our knowledge, this is the best published result. It achieves more than $100\boldsymbol {\times }$ speed-up on FCN layers over prior FPGA accelerators. An end-to-end evaluation with Caffe integration shows up to $29\boldsymbol {\times }$ and $150\boldsymbol {\times }$ performance and energy gains over Caffe on a 12-core Xeon server, and $5.7\boldsymbol {\times }$ better energy efficiency over the GPU implementation. Performance projections for a system with a high-end FPGA (Virtex7 690t) show even higher gains.

Caffeine: Towards Uniformed Representation and Acceleration for Deep Convolutional Neural Networks.

Caffeine: Toward Uniformed Representation and Acceleration for Deep Convolutional Neural Networks

Caffeinated FPGAs: FPGA Framework For Convolutional Neural Networks

FPGA‐accelerated Deep Convolutional Neural Networks for High Throughput and Energy Efficiency

A Scalable FPGA Accelerator for Convolutional Neural Networks.

Toward Full-Stack Acceleration of Deep Convolutional Neural Networks on FPGAs

A Convolutional Neural Network Accelerator Architecture with Fine-Granular Mixed Precision Configurability.

Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks

Throughput-Optimized FPGA Accelerator for Deep Convolutional Neural Networks

FeCaffe: FPGA-enabled Caffe with OpenCL for Deep Learning Training and Inference on Intel Stratix 10

A FPGA-based end-to-end acceleration framework for fast deployment of Convolutional Neural Networks

FPGA-based Accelerator for Convolutional Neural Network

An FPGA-Based CNN Accelerator Integrating Depthwise Separable Convolution

Acceleration of Deep Neural Network Training Using Field Programmable Gate Arrays

Exploring Heterogeneous Algorithms for Accelerating Deep Convolutional Neural Networks on FPGAs

A High Performance FPGA-based Accelerator for Large-Scale Convolutional Neural Networks

CaFPGA: an Automatic Generation Model for CNN Accelerator.

A Data-Center FPGA Acceleration Platform for Convolutional Neural Networks

An Efficient Hardware Accelerator for Structured Sparse Convolutional Neural Networks on FPGAs

An Efficient Design Flow for Accelerating Complicated-connected CNNs on a Multi-FPGA Platform.

FPGA-Based High-Throughput CNN Hardware Accelerator With High Computing Resource Utilization Ratio