Abstract:To efficiently deploy state-of-the-art deep neural network (DNN) workloads with growing computational intensity and structural complexity, scalable DNN accelerators have been proposed in recent years, which are featured by multitensor engines and distributed on-chip buffers. Such spatial architectures have significantly expanded scheduling space in terms of parallelism and data reuse potentials, which demands for delicate workload orchestration. Previous works on DNN's hardware mapping problem mainly focus on operator-level loop transformation for single array, which are insufficient for this new challenge. Resource partitioning methods for multi-engines such as CNN-partition and inter-layer pipelining have been studied. However, their intrinsic disadvantages of workload unbalance and pipeline delay still prevent scalable accelerators from releasing full potentials. In this paper, we propose atomic dataflow, a novel graph-level scheduling and mapping approach developed for DNN inference. Instead of partitioning hardware resources into fixed regions and binding each DNN layer to a certain region sequentially, atomic dataflow schedules the DNN computation graph in workload-specific granularity (atoms) to ensure PE-array utilization, supports flexible atom ordering to exploit parallelism, and orchestrates atom-engine mapping to optimize data reuse between spatially connected tensor engines. Firstly, we propose a simulated annealing based atomic tensor generation algorithm to minimize load unbalance. Secondly, we develop a dynamic programming based atomic DAG scheduling algorithm to systematically explore massive ordering potentials. Finally, to facilitate data locality and reduce expensive off-chip memory access, we present mapping and buffering strategies to efficiently utilize distributed on-chip storage. With an automated optimization framework being established, experimental results show significant improvements over baseline approaches in terms of performance, hardware utilization, and energy consumption.

Extending the Nested Parallel Model to the Nested Dataflow Model with Provably Efficient Schedulers

NestedMP: Taming Complex Configuration Space of Degree of Parallelism for Nested-Parallel Programs

Efficient Multi-Processor Scheduling in Increasingly Realistic Models

NestedMP: Enabling Cache-Aware Thread Mapping for Nested Parallel Shared Memory Applications

Parallelizing stateful operators in a distributed stream processing system: how, should you and how much?

Speculative Parallelization Using State Separation and Multiple Value Prediction.

Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learning

Atomic Dataflow based Graph-Level Workload Orchestration for Scalable DNN Accelerators

A Parallel Dynamic Programming Algorithm on a Multi-Core Architecture

Improving Performance of Dynamic Programming Via Parallelism and Locality on Multicore Architectures

Enhanced Parallelization via Constraints

Supporting Speculative Parallelization in the Presence of Dynamic Data Structures

On the Extreme Parallelism Inside Next-Generation Network Processors

POSTER: Provably Efficient Scheduling of Cache-Oblivious Wavefront Algorithms

Compile-Time Scheduling Of Multithread With Data Localities On Multiple Vector Processors

Speculative Parallelization of Sequential Loops on Multicores

Non-clairvoyant Weighted Flow Time Scheduling on Different Multi-processor Models

Parallel Batch Scheduling With Incompatible Job Families Via Constraint Programming

Interstellar: Using Halide's Scheduling Language to Analyze DNN Accelerators

A Nested Partitioning Scheme for Parallel Heterogeneous Clusters

The Potential of Synergistic Static, Dynamic and Speculative Loop Nest Optimizations for Automatic Parallelization