Abstract:Facing the growing complexity of Deep Neural Networks (DNNs), high-performance and power-efficient AI accelerators are desired to provide effective and affordable cloud inference services. We introduce our flagship product, i.e., the Cloudblazer i20 accelerator, which integrates the innovated Deep Thinking Unit (DTU 2.0). The design is driven by requests drawn from various AI inference applications and insights learned from our previous products. With careful tradeoffs in hardware-software co-design, Cloudblazer i20 delivers impressive performance and energy efficiency while maintaining acceptable hardware costs and software complexity/flexibility. To tackle computation- and data-intensive workloads, DTU 2.0 integrates powerful vector/matrix engines and a large-capacity multi-level memory hierarchy with high bandwidth. It supports comprehensive data flow and synchronization patterns to fully exploit parallelism in computation/memory access within or among concurrent tasks. Moreover, it enables sparse data compression/decompression, data broadcasting, repeated data transfer, and kernel code prefetching to optimize bandwidth utilization and reduce data access overheads. To utilize the underlying hardware and simplify the development of customized DNNs/operators, the software stack enables automatic optimizations (such as operator fusion and data flow tuning) and provides diverse programming interfaces for developers. Lastly, the energy consumption is optimized through dynamic power integrity and efficiency management, eliminating integrity risks and energy wastes. Based on the performance requirement, developers also can assign their workloads with the entire or partial hardware resources accordingly. Evaluated with 10 representative DNN models widely adopted in various domains, Cloudblazer i20 outperforms Nvidia T4 and A10 GPUs with a geometric mean of 2.22x and 1.16x in performance and 1.04x and 1.17x in energy efficiency, respectively. The improvements demonstrate the effectiveness of Cloudblazer i20’s design that emphasizes performance, efficiency, and flexibility.

Research on Parallel Acceleration for Deep Learning Inference Based on Many-Core ARM Platform.

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

High performance and energy efficient inference for deep learning on ARM processors

ParaX: boosting deep learning for big data analytics on many-core CPUs

Hardware Accelerated Optimization of Deep Learning Model on Artificial Intelligence Chip

A deep learning image recognition framework accelerator based parallel computing

Deep Learning Inference on Heterogeneous Mobile Processors: Potentials and Pitfalls

CAP: Communication-aware Automated Parallelization for Deep Learning Inference on CMP Architectures

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

An Optimization Toolchain Design Of Deep Learning Deployment Based On Heterogeneous Computing Platform

Reaching for the Sky: Maximizing Deep Learning Inference Throughput on Edge Devices with AI Multi-Tenancy

High Performance and Power Efficient Accelerator for Cloud Inference

Research on Deep Learning Acceleration Technique

An architecture-level analysis on deep learning models for low-impact computations

Research on Convolutional Neural Network Inference Acceleration and Performance Optimization for Edge Intelligence

Performance and Power Evaluation of AI Accelerators for Training Deep Learning Models

Accelerating Large Scale Deep Learning Inference Through DeepCPU at Microsoft.

Optimizing Direct Convolutions on ARM Multi-Cores

CoDL: efficient CPU-GPU co-execution for deep learning inference on mobile devices

Joint Architecture Design and Workload Partitioning for DNN Inference on Industrial IoT Clusters

Transformer Inference Acceleration in Edge Computing Environment