Abstract:The Deep Learning (DL) community sees many novel topologies published each year. Achieving high performance on each new topology remains challenging, as each requires some level of manual effort. This issue is compounded by the proliferation of frameworks and hardware platforms. The current approach, which we call "direct optimization", requires deep changes within each framework to improve the training performance for each hardware backend (CPUs, GPUs, FPGAs, ASICs) and requires $\mathcal{O}(fp)$ effort; where $f$ is the number of frameworks and $p$ is the number of platforms. While optimized kernels for deep-learning primitives are provided via libraries like Intel Math Kernel Library for Deep Neural Networks (MKL-DNN), there are several compiler-inspired ways in which performance can be further optimized. Building on our experience creating neon (a fast deep learning library on GPUs), we developed Intel nGraph, a soon to be open-sourced C++ library to simplify the realization of optimized deep learning performance across frameworks and hardware platforms. Initially-supported frameworks include TensorFlow, MXNet, and Intel neon framework. Initial backends are Intel Architecture CPUs (CPU), the Intel(R) Nervana Neural Network Processor(R) (NNP), and NVIDIA GPUs. Currently supported compiler optimizations include efficient memory management and data layout abstraction. In this paper, we describe our overall architecture and its core components. In the future, we envision extending nGraph API support to a wider range of frameworks, hardware (including FPGAs and ASICs), and compiler optimizations (training versus inference optimizations, multi-node and multi-device scaling via efficient sub-graph partitioning, and HW-specific compounding of operations).

NEST‐C: A deep learning compiler framework for heterogeneous computing systems with artificial intelligence accelerators

A Near Memory Computing FPGA Architecture for Neural Network Acceleration

Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning

An Optimization Toolchain Design Of Deep Learning Deployment Based On Heterogeneous Computing Platform

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

PowerFusion: A Tensor Compiler with Explicit Data Movement Description and Instruction-level Graph IR

Survey and design of paleozoic: a high-performance compiler tool chain for deep learning inference accelerator

A fine-grained mixed precision DNN accelerator using a two-stage big-little core RISC-V MCU.

Restoring the Broken Covenant Between Compilers and Deep Learning Accelerators

gem5-NVDLA: A Simulation Framework for Compiling, Scheduling and Architecture Evaluation on AI System-on-Chips

A Metaprogramming and Autotuning Framework for Deploying Deep Learning Applications

A General-Purpose Compiler Design for Instruction-Based AI Accelerator Implementation.

AI Powered Compiler Techniques for DL Code Optimization

NNest: Early-Stage Design Space Exploration Tool for Neural Network Inference Accelerators

Efficient Hardware Optimization Strategies For Deep Neural Networks Acceleration Chip

An In-depth Comparison of Compilers for Deep Neural Networks on Hardware

Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference

oneDNN Graph Compiler: A Hybrid Approach for High-Performance Deep Learning Compilation

An Automated Compiler for RISC-V Based DNN Accelerator

Evaluating Emerging AI/ML Accelerators: IPU, RDU, and NVIDIA/AMD GPUs

A Heterogeneous Full-stack AI Platform for Performance Monitoring and Hardware-specific Optimizations