Abstract:Supernet training, a prevalent and important paradigm in Neural Architecture Search, embeds the whole DNN architecture search space into one monolithic supernet, iteratively activates a subset of the supernet (i.e., a subnet) for fitting each batch of data, and searches a high-quality subnet which meets specific requirements. Although training subnets in parallel on multiple GPUs is desirable for acceleration, there inherently exists a race hazard that concurrent subnets may access the same DNN layers. Existing systems support neither efficiently parallelizing subnets’ training executions, nor resolving the race hazard deterministically, leading to unreproducible training procedures and potentiallly non-trivial accuracy loss. We present NASPipe, the first high-performance and reproducible distributed supernet training system via causal synchronous parallel (CSP) pipeline scheduling abstraction: NASPipe partitions a supernet across GPUs and concurrently executes multiple generated sub-tasks (subnets) in a pipelined manner; meanwhile, it oversees the correlations between the subnets and deterministically resolves any causal dependency caused by subnets’ layer sharing. To obtain high performance, NASPipe’s CSP scheduler exploits the fact that the larger a supernet spans, the fewer dependencies manifest between chronologically close subnets; therefore, it aggressively schedules the subnets with larger chronological orders into execution, only if they are not causally dependent on unfinished precedent subnets. Moreover, to relieve the excessive GPU memory burden for holding the whole supernet’s parameters, NASPipe uses a context switch technique that stashes the whole supernet in CPU memory, precisely predicts the subnets’ schedule, and pre-fetches/evicts a subnet before/after its execution. The evaluation shows that NASPipe is the only system that retains supernet training reproducibility, while achieving a comparable and even higher performance (up to 7.8X) compared to three recent pipeline training systems (e.g., GPipe).

Choice of Parallelism: Multi-GPU Driven Pipeline for Huge Academic Backbone Network

Efficient Modeling and Real-Time Rendering of Massive Urban Pipelines Based on GPU

Large-Scale Datasets Visualization on Networks Using Dynamic Pipeline Decomposition and Mapping

GPU Computations on Hadoop Clusters for Massive Data Processing

PipePar: Enabling fast DNN pipeline parallel training in heterogeneous GPU clusters

Optimizing execution for pipelined‐based distributed deep learning in a heterogeneously networked GPU cluster

A GPU-accelerated Parallel Network Traffic Analysis System

Large Scale Multi-GPU Based Parallel Traffic Simulation for Accelerated Traffic Assignment and Propagation

GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism

Pipeline Parallelism for Inference on Heterogeneous Edge Computing

Apache Spark Accelerated Deep Learning Inference for Large Scale Satellite Image Analytics

Hybrid CPU-GPU Framework for Network Motifs

GPU-accelerated hydraulic simulations of large-scale natural gas pipeline networks based on a two-level parallel process

Analyzing the Performance of Graph Neural Networks with Pipe Parallelism

Design and Optimization of a Big Data Computing Framework Based on CPU/GPU Cluster

NASPipe: high performance and reproducible pipeline parallel supernet training via causal synchronous parallelism

Remote visualization for large terrain surfaces based on parallel streaming pipeline architecture

Implementation of MapReduce parallel computing framework based on multi-data fusion sensors and GPU cluster

Accelerating Large Scale Artificial Society Simulation with Cpu/Gpu Based Heterogeneous Parallel Method