Abstract:With the recent advancement of multilayer convolutional neural networks (CNNs) and fully connected networks (FCNs), deep learning has achieved amazing success in many areas, especially in visual content understanding and classification. To improve the performance and energy efficiency of the computation-demanding CNN, the FPGA-based acceleration emerges as one of the most attractive alternatives. In this paper, we design and implement Caffeine, a hardware/software co-designed library to efficiently accelerate the entire CNN and FCN on FPGAs. First, we propose a uniformed convolutional matrix-multiplication representation for both computation-bound convolutional layers and communication-bound FCN layers. Based on this representation, we optimize the accelerator micro-architecture and maximize the underlying FPGA computing and bandwidth resource utilization based on a revised roofline model. Moreover, we design an automation flow to directly compile high-level network definitions to the final FPGA accelerator. As a case study, we integrate Caffeine into the industry-standard software deep learning framework Caffe. We evaluate Caffeine and its integration with Caffe by implementing VGG16 and AlexNet networks on multiple FPGA platforms. Caffeine achieves a peak performance of 1460 giga fixed point operations per second on a medium-sized Xilinx KU060 FPGA board; to our knowledge, this is the best published result. It achieves more than $100\boldsymbol {\times }$ speed-up on FCN layers over prior FPGA accelerators. An end-to-end evaluation with Caffe integration shows up to $29\boldsymbol {\times }$ and $150\boldsymbol {\times }$ performance and energy gains over Caffe on a 12-core Xeon server, and $5.7\boldsymbol {\times }$ better energy efficiency over the GPU implementation. Performance projections for a system with a high-end FPGA (Virtex7 690t) show even higher gains.

Energy-Efficient Cnn Implementation on A Deeply Pipelined Fpga Cluster

Maximizing CNN Throughput on FPGA Clusters.

Improving CNN Performance on FPGA Clusters Through Topology Exploration

Designing Deep Learning Hardware Accelerator and Efficiency Evaluation

UniCNN: A Pipelined Accelerator Towards Uniformed Computing for CNNs

An Efficient Design Flow for Accelerating Complicated-connected CNNs on a Multi-FPGA Platform.

A Memory-Optimized and Energy-Efficient CNN Acceleration Architecture Based on FPGA.

An FPGA-Based CNN Accelerator Integrating Depthwise Separable Convolution

An Energy-Efficient FPGA-Based Embedded System for CNN Application

Toward Full-Stack Acceleration of Deep Convolutional Neural Networks on FPGAs

PipeCNN: An OpenCL-Based FPGA Accelerator for Large-Scale Convolution Neuron Networks

FPDeep: Scalable Acceleration of CNN Training on Deeply-Pipelined FPGA Clusters

Communication-Aware and Resource-Efficient NoC-Based Architecture for CNN Acceleration

A Power-Efficient and High Performance FPGA Accelerator for Convolutional Neural Networks: Work-in-progress.

FPGA‐accelerated Deep Convolutional Neural Networks for High Throughput and Energy Efficiency

Full-stack Optimization for Accelerating CNNs with FPGA Validation

Acceleration of Deep Neural Network Training Using Field Programmable Gate Arrays

FPGA-Based High-Throughput CNN Hardware Accelerator With High Computing Resource Utilization Ratio

Energy-Efficient And High-Throughput Fpga-Based Accelerator For Convolutional Neural Networks

Caffeine: Towards Uniformed Representation and Acceleration for Deep Convolutional Neural Networks.

Caffeine: Toward Uniformed Representation and Acceleration for Deep Convolutional Neural Networks