Abstract:Convolutional neural networks (CNNs) are emerging as powerful tools for image processing in important commercial applications. We focus on the important problem of improving the latency of image recognition. While CNNs are highly amenable to prefetching and multithreading to avoid memory latency issues, CNNs’ large data – each layer’s input, filters, and output – poses a memory bandwidth problem. While previous work captures only some of the enormous data reuse, full reuse implies that the initial input image and filters are read once from off-chip and the final output is written once off-chip without spilling the intermediate layers’ data to off-chip. We propose Occam to capture full reuse via four contributions. First, we identify the necessary conditions for full reuse. Second, we identify the dependence closure as the sufficient condition to capture full reuse using the least on-chip memory. Third, because the dependence closure is often too large to fit in on-chip memory, we propose a dynamic programming algorithm that optimally partitions a given CNN to guarantee the least off-chip traffic at the partition boundaries for a given on-chip capacity. While tiling is well-known, our contribution determines the optimal cross-layer tiles. Occam’s partitions reside on different chips, forming a pipeline so that a partition’s filters and dependence closure remain on-chip as different images pass through (i.e., each partition incurs off-chip traffic only for its inputs and outputs). Finally, because the optimal partitions may result in an unbalanced pipeline, we propose staggered asynchronous pipelines (STAPs) that replicate bottleneck stages to improve throughput by staggering mini-batches across replicas. Importantly, STAPs achieve balanced pipelines without changing Occam’s optimal partitioning. Our simulations show that, on average, Occam cuts off-chip transfers by 21× and achieves 2.04× and 1.21× better performance, and 33% better energy than the base case, respectively. Using a field-programmable gate array (FPGA) implementation, Occam performs 6.1× and 1.5× better, on average, than the base case and Layer Fusion, respectively.

FreeLunch: Compression-based GPU Memory Management for Convolutional Neural Networks

Memory-Efficient Compression Based on Least-Squares Fitting in Convolutional Neural Network Accelerators.

Efficient Memory Management for GPU-based Deep Learning Systems

LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reduction

A Novel Memory-Scheduling Strategy for Large Convolutional Neural Network on Memory-Limited Devices

NeuroFlux: Memory-Efficient CNN Training Using Adaptive Local Learning

Accelerating Tensor Swapping in GPUs with Self-Tuning Compression

High Performance CNN Accelerators Based on Hardware and Algorithm Co-Optimization

Improved JPEG Lossless Compression for Compression of Intermediate Layers in Neural Networks Based on Compute-In-Memory

Memory-Efficient CNN Accelerator Based on Interlayer Feature Map Compression

Less Memory Means smaller GPUs: Backpropagation with Compressed Activations

An Efficient CNN Inference Accelerator Based on Intra- and Inter-Channel Feature Map Compression

CSWAP: A Self-Tuning Compression Framework for Accelerating Tensor Swapping in GPUs

Conv-inheritance: A hardware-efficient method to compress convolutional neural networks for edge applications

Relative Indexed Compressed Sparse Filter Encoding Format for Hardware-Oriented Acceleration of Deep Convolutional Neural Networks

High Throughput CNN Inference and Training with In-Cache Computation

Occam: Optimal Data Reuse for Convolutional Neural Networks

A Computing Efficient Hardware Architecture for Sparse Deep Neural Network Computing

Optimization of GPU Memory Usage for Training Deep Neural Networks.

Mitigating Memory Wall Effects in CNN Engines with On-the-Fly Weights Generation

Optimally Scheduling CNN Convolutions for Efficient Memory Access