Abstract:Since the breakthrough performance of AlexNet in 2012, convolutional neural networks (convnets) have grown into extremely powerful vision models. Deep learning researchers have used convnets to perform vision tasks with accuracy that was unachievable a decade ago. Confronted with the immense computation that convnets use, deep learning researchers also became interested in efficiency. However, the engineers who deployed efficient convnets soon realized that they were slower than the previous generation, despite using fewer operations. Many reverted to older models that ran faster. Hence researchers switched the objective of their search from arithmetic complexity to latency and produced a new wave of models that performed better. Paradoxically, these models also used more operations. Skepticism grew among researchers and engineers alike about the relevance of arithmetic complexity. Contrary to the prevailing view that latency and arithmetic complexity are irreconcilable, a simple formula relates both through computational efficiency. This insight enabled us to co-optimize the separate factors that determine latency. We observed that the degenerate conv2d layers that produce the best accuracy--complexity trade-off also use significant memory resources and have low computational efficiency. We devised block fusion algorithms to implement all the layers of a residual block in a single kernel, thereby creating temporal locality, avoiding communication, and reducing workspace size. Our ConvFirst model with block-fusion kernels has less arithmetic complexity and greater computational efficiency than baseline models and kernels, and ran approximately four times as fast as ConvNeXt. We also created novel tools, including efficiency gap plots and waterline analysis. Our unified approach to convnet efficiency envisions a new era of models and kernels that achieve greater accuracy at lower cost.

Faster Convolution Inference Through Using Pre-Calculated Lookup Tables

TabConv: Low-Computation CNN Inference via Table Lookups

Optical Convolution Based Computational Method for Low-Power Image Processing

Look-ups are not (yet) all you need for deep learning inference

Algebraic Representations for Faster Predictions in Convolutional Neural Networks

Shift-based Primitives for Efficient Convolutional Neural Networks

AddressNet: Shift-Based Primitives for Efficient Convolutional Neural Networks.

Efficient Convolutional Neural Networks Utilizing Fine-Grained Fast Fourier Transforms

Deep Convolutional Tables: Deep Learning without Convolutions

Accelerating Convolutional Neural Network Inference Based on a Reconfigurable Sliced Systolic Array

ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networks

cuConv: A CUDA Implementation of Convolution for CNN Inference

ILP-M Conv: Optimize Convolution Algorithm for Single-Image Convolution Neural Network Inference on Mobile GPUs

Subtractor-Based CNN Inference Accelerator

Efficient and Accurate Approximations of Nonlinear Convolutional Networks

Comb Convolution for Efficient Convolutional Architecture

FTConv: FPGA Acceleration for Transposed Convolution Layers in Deep Neural Networks

Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers

On the Efficiency of Convolutional Neural Networks

A Pre-defined Sparse Kernel Based Convolution for Deep CNNs

More is Less: A More Complicated Network with Less Inference Complexity.