Abstract:A cost-effective multi-tenant neural network execution is becoming one of the most important design goals for modern neural network accelerators. For example, as emerging AI services consist of many heterogeneous neural network executions, a cloud provider wants to serve a large number of clients using a single AI accelerator for improving its cost effectiveness. Therefore, an ideal next-generation neural network accelerator should support a simultaneous multi-neural network execution, while fully utilizing its hardware resources. However, existing accelerators which are optimized for a single neural network execution can suffer from severe resource underutilization when running multiple neural networks, mainly due to the load imbalance between computation and memory-access tasks from different neural networks. In this paper, we propose AI-MultiTasking (AI-MT), a novel accelerator architecture which enables a cost-effective, high-performance multi-neural network execution. The key idea of AI-MT is to fully utilize the accelerator's computation resources and memory bandwidth by matching compute- and memory-intensive tasks from different networks and executing them in parallel. However, it is highly challenging to find and schedule the best load-matching tasks from different neural networks during runtime, without significantly increasing the size of on-chip memory. To overcome the challenges, AI-MT first creates fine-grain tasks at compile time by dividing each layer into multiple identical sub-layers. During runtime, AI-MT dynamically applies three sub-layer scheduling methods: memory block prefetching and compute block merging for the best resource load matching, and memory block eviction for the minimum on-chip memory footprint. Our evaluations using MLPerf benchmarks show that AI-MT achieves up to 1.57x speedup over the baseline scheduling method.

A neural network accelerator for mobile application processors

A Near Memory Computing FPGA Architecture for Neural Network Acceleration

A design framework for processing-in-memory accelerator

A Convolutional Neural Network Accelerator Architecture with Fine-Granular Mixed Precision Configurability.

DaDianNao: A Machine-Learning Supercomputer

A Novel Throughput Enhancement Method for Deep Learning Applications on Mobile Devices With Heterogeneous Processors

A Data-Driven Asynchronous Neural Network Accelerator

An Overview of Energy-Efficient Hardware Accelerators for On-Device Deep-Neural-Network Training

Efficient Hardware Optimization Strategies For Deep Neural Networks Acceleration Chip

A Small-Footprint Accelerator for Large-Scale Neural Networks

A 3D Tiled Low Power Accelerator for Convolutional Neural Network

Reconfigurable Binary Neural Network Accelerator with Adaptive Parallelism Scheme

Energy-Efficient Accelerator Design for Deformable Convolution Networks

Design of a Convolutional Neural Network Accelerator Based on On-Chip Data Reordering

A Multi-Neural Network Acceleration Architecture

Improving Efficiency in Neural Network Accelerator Using Operands Hamming Distance Optimization

A Construction Kit for Efficient Low Power Neural Network Accelerator Designs

A Low-Power Accelerator for Deep Neural Networks with Enlarged Near-Zero Sparsity

Heterogeneous Multi-core Array-based DNN Accelerator

Software-defined Design Space Exploration for an Efficient DNN Accelerator Architecture