Abstract:Machine learning models with various tensor operators are becoming ubiquitous in recent years. There are two types of operators in machine learning: compute-intensive operators (e.g., GEMM and convolution) and memory-intensive operators (e.g., ReLU and softmax). In emerging machine learning models, compute-intensive operators are usually organized in a chain structure. With the continual specialization of hardware, the gap between computing performance and memory bandwidth has become more prominent. Consequently, the implementations of many compute-intensive operator chains are bounded by memory bandwidth, and generating fused kernels to improve locality for these compute-intensive operators becomes necessary. But in existing machine learning compilers, there lack both precise analysis and efficient optimization for compute-intensive operator chains on different accelerators. As a result, they usually produce sub-optimal performance for these operator chains.In this paper, we propose Chimera, an optimizing framework that can efficiently improve the locality of compute-intensive operator chains on different hardware accelerators. In Chimera, each compute-intensive operator is composed of a series of computation blocks. To generate efficient fused kernels for the operator chains, optimizations for both inter-block and intra-block are required. For inter-block optimization, Chimera decides the optimized block execution order by minimizing the data movement volume among blocks using an analytical model. For intra-block optimization, Chimera uses unified replaceable micro kernels to apply hardware-specific optimizations for different accelerators. Finally, Chimera generates fused kernels for compute-intensive operator chains. Evaluation of batch GEMM chains and convolution chains on CPU, GPU, and NPU shows that Chimera achieves up to 2.87×, 2.29×, and 2.39× speedups to hand-tuned libraries. Compared to state-of-the-art compilers, the speedups are up to 2.29×, 1.64×, and 1.14× for CPU, GPU, and NPU.

Weave: Abstraction and Integration Flow for Accelerators of Generated Modules

A design framework for processing-in-memory accelerator

An accelerator-aware microarchitecture simulator for design space exploration

Early DSE and Automatic Generation of Coarse Grained Merged Accelerators

WELDER: Scheduling Deep Learning Memory Access Via Tile-graph

Allo: A Programming Model for Composable Accelerator Design

AutoAccel: Automated Accelerator Generation and Optimization with Composable, Parallel and Pipeline Architecture

gem5-NVDLA: A Simulation Framework for Compiling, Scheduling and Architecture Evaluation on AI System-on-Chips

Survey and design of paleozoic: a high-performance compiler tool chain for deep learning inference accelerator

Bring Your Own Codegen to Deep Learning Compiler

Towards Dataflow-Based Graph Accelerator.

A General-Purpose Compiler Design for Instruction-Based AI Accelerator Implementation.

LoopTree: Exploring the Fused-layer Dataflow Accelerator Design Space

Compiling Halide Programs to Push-Memory Accelerators

PowerFusion: A Tensor Compiler with Explicit Data Movement Description and Instruction-level Graph IR

TileFlow: A Framework for Modeling Fusion Dataflow Via Tree-based Analysis.

Mrna: Enabling Efficient Mapping Space Exploration for a Reconfiguration Neural Accelerator

Understanding Reuse, Performance, and Hardware Cost of DNN Dataflows: A Data-Centric Approach Using MAESTRO

Monad: Towards Cost-effective Specialization for Chiplet-based Spatial Accelerators

PALM: A Efficient Performance Simulator for Tiled Accelerators with Large-scale Model Training

Chimera: an Analytical Optimizing Framework for Effective Compute-intensive Operators Fusion