Abstract:Deep neural networks (DNNs) are increasingly deployed in various image recognition and natural language processing applications. The continuous demand for accuracy and high performance has led to innovations in DNN design and a proliferation of new operators. However, existing DNN training frameworks such as PyTorch and TensorFlow only support a limited range of operators and rely on hand-optimized libraries to provide efficient implementations for these operators. To evaluate novel neural networks with new operators, the programmers have to either replace the holistic new operators with existing operators or provide low-level implementations manually. Therefore, a critical requirement for DNN training frameworks is to provide high-performance implementations for the neural networks containing new operators automatically in the absence of efficient library support. In this article, we introduce NeoFlow, which is a flexible framework for enabling efficient compilation for high-performance DNN training. NeoFlow allows the programmers to directly write customized expressions as new operators to be mapped to graph representation and low-level implementations automatically, providing both high programming productivity and high performance. First, NeoFlow provides expression-based automatic differentiation to support customized model definitions with new operators. Then, NeoFlow proposes an efficient compilation system that partitions the neural network graph into subgraphs, explores optimized schedules, and generates high-performance libraries for subgraphs automatically. Finally, NeoFlow develops an efficient runtime system to combine the compilation and training as a whole by overlapping their execution. In the experiments, we examine the numerical accuracy and performance of NeoFlow. The results show that NeoFlow can achieve similar or even better performance at the operator and whole graph level for DNNs compared to deep learning frameworks. Especially, for novel networks training, the geometric mean speedups of NeoFlow to PyTorch, TensorFlow, and CuDNN are 3.16X, 2.43X, and 1.92X, respectively.

Uncovering Nested Data Parallelism and Data Reuse in DNN Computation with FractalTensor

Accelerating Sparse Graph Neural Networks with Tensor Core Optimization

TANGRAM: Optimized Coarse-Grained Dataflow for Scalable NN Accelerators

TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism

TSTC: Enabling Efficient Training Via Structured Sparse Tensor Compilation

NeoFlow: A Flexible Framework for Enabling Efficient Compilation for High Performance DNN Training

ETO: Accelerating Optimization of DNN Operators by High-Performance Tensor Program Reuse

Exploiting Potential of Deep Neural Networks by Layer-Wise Fine-Grained Parallelism

EINNET: Optimizing Tensor Programs with Derivation-Based Transformations.

Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions

TAP: Accelerating Large-Scale DNN Training Through Tensor Automatic Parallelisation

Optimizing DNNs with Partially Equivalent Transformations and Automated Corrections

High-Performance Tensor-Train Primitives Using GPU Tensor Cores

PowerFusion: A Tensor Compiler with Explicit Data Movement Description and Instruction-level Graph IR

Optimizing Tensor Computation Graphs with Equality Saturation and Monte Carlo Tree Search

Realistic Acceleration of Neural Networks with Fine-Grained Tensor Decomposition

Lftf: A Framework for Efficient Tensor Analytics at Scale

Abstracting Sparse DNN Acceleration via Structured Sparse Tensor Decomposition

DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusion

DSTC: Dual-Side Sparsity Tensor Core for DNNs Acceleration on Modern GPU Architectures

DF-GNN: Dynamic Fusion Framework for Attention Graph Neural Networks on GPUs