Abstract:Deep neural networks (DNNs) are increasingly deployed in various image recognition and natural language processing applications. The continuous demand for accuracy and high performance has led to innovations in DNN design and a proliferation of new operators. However, existing DNN training frameworks such as PyTorch and TensorFlow only support a limited range of operators and rely on hand-optimized libraries to provide efficient implementations for these operators. To evaluate novel neural networks with new operators, the programmers have to either replace the holistic new operators with existing operators or provide low-level implementations manually. Therefore, a critical requirement for DNN training frameworks is to provide high-performance implementations for the neural networks containing new operators automatically in the absence of efficient library support. In this article, we introduce NeoFlow, which is a flexible framework for enabling efficient compilation for high-performance DNN training. NeoFlow allows the programmers to directly write customized expressions as new operators to be mapped to graph representation and low-level implementations automatically, providing both high programming productivity and high performance. First, NeoFlow provides expression-based automatic differentiation to support customized model definitions with new operators. Then, NeoFlow proposes an efficient compilation system that partitions the neural network graph into subgraphs, explores optimized schedules, and generates high-performance libraries for subgraphs automatically. Finally, NeoFlow develops an efficient runtime system to combine the compilation and training as a whole by overlapping their execution. In the experiments, we examine the numerical accuracy and performance of NeoFlow. The results show that NeoFlow can achieve similar or even better performance at the operator and whole graph level for DNNs compared to deep learning frameworks. Especially, for novel networks training, the geometric mean speedups of NeoFlow to PyTorch, TensorFlow, and CuDNN are 3.16X, 2.43X, and 1.92X, respectively.

Jittor: a Novel Deep Learning Framework with Meta-Operators and Unified Graph Execution

oneDNN Graph Compiler: A Hybrid Approach for High-Performance Deep Learning Compilation

An Optimization Toolchain Design Of Deep Learning Deployment Based On Heterogeneous Computing Platform

Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning

JANUS: Fast and Flexible Deep Learning via Symbolic Graph Execution of Imperative Programs

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

ALT: Boosting Deep Learning Performance by Breaking the Wall between Graph and Operator Level Optimizations

JITGNN: A deep graph neural network framework for Just-In-Time bug prediction

NeoFlow: A Flexible Framework for Enabling Efficient Compilation for High Performance DNN Training

MAGPY: Compiling Eager Mode DNN Programs by Monitoring Execution States.

AutoGraph: Imperative-style Coding with Graph-based Performance

Visualizing JIT Compiler Graphs

No Saved Kaleidosope: an 100% Jitted Neural Network Coding Language with Pythonic Syntax

EasyView: Enabling and Scheduling Tensor Views in Deep Learning Compilers

ALT: Breaking the Wall Between Data Layout and Loop Optimizations for Deep Learning Compilation

A Composable Just-In-Time Programming Framework with LLMs and FBP

Bring Your Own Codegen to Deep Learning Compiler

Enabling Tensor Language Model to Assist in Generating High-Performance Tensor Programs for Deep Learning.

AI Powered Compiler Techniques for DL Code Optimization

ETO: Accelerating Optimization of DNN Operators by High-Performance Tensor Program Reuse

PowerFusion: A Tensor Compiler with Explicit Data Movement Description and Instruction-level Graph IR