Abstract:A cost-effective multi-tenant neural network execution is becoming one of the most important design goals for modern neural network accelerators. For example, as emerging AI services consist of many heterogeneous neural network executions, a cloud provider wants to serve a large number of clients using a single AI accelerator for improving its cost effectiveness. Therefore, an ideal next-generation neural network accelerator should support a simultaneous multi-neural network execution, while fully utilizing its hardware resources. However, existing accelerators which are optimized for a single neural network execution can suffer from severe resource underutilization when running multiple neural networks, mainly due to the load imbalance between computation and memory-access tasks from different neural networks. In this paper, we propose AI-MultiTasking (AI-MT), a novel accelerator architecture which enables a cost-effective, high-performance multi-neural network execution. The key idea of AI-MT is to fully utilize the accelerator's computation resources and memory bandwidth by matching compute- and memory-intensive tasks from different networks and executing them in parallel. However, it is highly challenging to find and schedule the best load-matching tasks from different neural networks during runtime, without significantly increasing the size of on-chip memory. To overcome the challenges, AI-MT first creates fine-grain tasks at compile time by dividing each layer into multiple identical sub-layers. During runtime, AI-MT dynamically applies three sub-layer scheduling methods: memory block prefetching and compute block merging for the best resource load matching, and memory block eviction for the minimum on-chip memory footprint. Our evaluations using MLPerf benchmarks show that AI-MT achieves up to 1.57x speedup over the baseline scheduling method.

DasNet: Dynamic Adaptive Structure for Accelerating Multi-task Convolutional Neural Network

MLCNN: Cross-Layer Cooperative Optimization and Accelerator Architecture for Speeding Up Deep Learning Applications

SteppingNet: A Stepping Neural Network with Incremental Accuracy Enhancement

Acceleration of multi-task cascaded convolutional networks

Dynamic Multi-Task Learning with Convolutional Neural Network

Dynamic Multi-path Neural Network

ENAS4D: Efficient Multi-stage CNN Architecture Search for Dynamic Inference

Dynamic Deep Neural Network Inference Via Adaptive Channel Skipping.

Memory-Computing Decoupling: A DNN Multitasking Accelerator with Adaptive Data Arrangement.

Sparsing Deep Neural Network Using Semi-Discrete Matrix Decomposition

Fast CNN Inference by Adaptive Sparse Matrix Decomposition.

A Progressive Subnetwork Searching Framework for Dynamic Inference

Reducing structure of deep Convolutional Neural Networks for Huawei Accurate and Fast Mobile Video Annotation Challenge

Learning Versatile Neural Architectures by Propagating Network Codes

Multi-task Learning with Deformable Convolution

Automl For Densenet Compression

Multi-scale Adaptive Networks for Efficient Inference

A Multi-Neural Network Acceleration Architecture

Adaptive Feature Aggregation in Deep Multi-Task Convolutional Neural Networks

CO-Net: Learning Multiple Point Cloud Tasks at Once with A Cohesive Network

Cross-Task Affinity Learning for Multitask Dense Scene Predictions