Abstract:Facing the growing complexity of Deep Neural Networks (DNNs), high-performance and power-efficient AI accelerators are desired to provide effective and affordable cloud inference services. We introduce our flagship product, i.e., the Cloudblazer i20 accelerator, which integrates the innovated Deep Thinking Unit (DTU 2.0). The design is driven by requests drawn from various AI inference applications and insights learned from our previous products. With careful tradeoffs in hardware-software co-design, Cloudblazer i20 delivers impressive performance and energy efficiency while maintaining acceptable hardware costs and software complexity/flexibility. To tackle computation- and data-intensive workloads, DTU 2.0 integrates powerful vector/matrix engines and a large-capacity multi-level memory hierarchy with high bandwidth. It supports comprehensive data flow and synchronization patterns to fully exploit parallelism in computation/memory access within or among concurrent tasks. Moreover, it enables sparse data compression/decompression, data broadcasting, repeated data transfer, and kernel code prefetching to optimize bandwidth utilization and reduce data access overheads. To utilize the underlying hardware and simplify the development of customized DNNs/operators, the software stack enables automatic optimizations (such as operator fusion and data flow tuning) and provides diverse programming interfaces for developers. Lastly, the energy consumption is optimized through dynamic power integrity and efficiency management, eliminating integrity risks and energy wastes. Based on the performance requirement, developers also can assign their workloads with the entire or partial hardware resources accordingly. Evaluated with 10 representative DNN models widely adopted in various domains, Cloudblazer i20 outperforms Nvidia T4 and A10 GPUs with a geometric mean of 2.22x and 1.16x in performance and 1.04x and 1.17x in energy efficiency, respectively. The improvements demonstrate the effectiveness of Cloudblazer i20’s design that emphasizes performance, efficiency, and flexibility.

A Scalable Multi-TeraOPS Core for AI Training and Inference

A Scalable Multi-TeraOPS Core for AI Training and Inference

DaDianNao: A Machine-Learning Supercomputer

A 7-nm Four-Core Mixed-Precision AI Chip With 26.2-TFLOPS Hybrid-FP8 Training, 104.9-TOPS INT4 Inference, and Workload-Aware Throttling

High Performance Scalable FPGA Accelerator for Deep Neural Networks

RaPiD: AI Accelerator for Ultra-low Precision Training and Inference

A 4.69-TOPS/W Training, 2.34- $\mu$ J/Image Inference On-Chip Training Accelerator with Inference-Compatible Backpropagation and Design Space Exploration in 28-Nm CMOS

A 28nm, 4.69TOPS/W Training, 2.34µj/lmage Inference, On-Chip Training Accelerator with Inference-compatible Back Propagation

A fine-grained mixed precision DNN accelerator using a two-stage big-little core RISC-V MCU.

HIPU: A Hybrid Intelligent Processing Unit With Fine-Grained ISA for Real-Time Deep Neural Network Inference Applications

7.2 A 12nm Programmable Convolution-Efficient Neural-Processing-Unit Chip Achieving 825TOPS

A Scalable Multi-Chiplet Deep Learning Accelerator with Hub-Side 2.5D Heterogeneous Integration.

120 GOPS Photonic Tensor Core in Thin-film Lithium Niobate for Inference and in-situ Training

Ifpna: A Flexible and Efficient Deep Neural Network Accelerator with a Programmable Data Flow Engine in 28nm CMOS.

PAICORE: A 1.9-Million-neuron 5.181-TSOPS/W Digital Neuromorphic Processor with Unified SNN-ANN and On-Chip Learning Paradigm

22.1 A 12.4TOPS/W @ 136GOPS AI-IoT System-on-Chip with 16 RISC-V, 2-to-8b Precision-Scalable DNN Acceleration and 30%-Boost Adaptive Body Biasing

Hypermultiplexed Integrated Tensor Optical Processor

High Performance and Power Efficient Accelerator for Cloud Inference

All-rounder: A flexible DNN accelerator with diverse data format support

A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference

Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10