Abstract:While maximizing deep neural networks' (DNNs') acceleration efficiency requires a joint search/design of three different yet highly coupled aspects, including the networks, bitwidths, and accelerators, the challenges associated with such a joint search have not yet been fully understood and addressed. The key challenges include (1) the dilemma of whether to explode the memory consumption due to the huge joint space or achieve sub-optimal designs, (2) the discrete nature of the accelerator design space that is coupled yet different from that of the networks and bitwidths, and (3) the chicken and egg problem associated with network-accelerator co-search, i.e., co-search requires operation-wise hardware cost, which is lacking during search as the optimal accelerator depending on the whole network is still unknown during search. To tackle these daunting challenges towards optimal and fast development of DNN accelerators, we propose a framework dubbed Auto-NBA to enable jointly searching for the Networks, Bitwidths, and Accelerators, by efficiently localizing the optimal design within the huge joint design space for each target dataset and acceleration specification. Our Auto-NBA integrates a heterogeneous sampling strategy to achieve unbiased search with constant memory consumption, and a novel joint-search pipeline equipped with a generic differentiable accelerator search engine. Extensive experiments and ablation studies validate that both Auto-NBA generated networks and accelerators consistently outperform state-of-the-art designs (including co-search/exploration techniques, hardware-aware NAS methods, and DNN accelerators), in terms of search time, task accuracy, and accelerator efficiency. Our codes are available at: https://github.com/RICE-EIC/Auto-NBA.

TAPS: Topology-Aware Intra-Operator Parallelism Strategy Searching Algorithm for Deep Neural Networks

TAP: Accelerating Large-Scale DNN Training Through Tensor Automatic Parallelisation

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming

3D Parallelism for Transformers Via Integer Programming

TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism

ETO: Accelerating Optimization of DNN Operators by High-Performance Tensor Program Reuse

PaSE: Parallelization Strategies for Efficient DNN Training

TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs

Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Nnscaler: Constraint-Guided Parallelization Plan Generation for Deep Learning Training.

IOS: Inter-Operator Scheduler for CNN Acceleration

Learning Partial Differential Equations with Deep Parallel Neural Operator

Optimization of Topology-Aware Job Allocation on a High-Performance Computing Cluster by Neural Simulated Annealing

Ansor : Generating High-Performance Tensor Programs for Deep Learning

FTSO: Effective NAS via First Topology Second Operator

BaPipe: Exploration of Balanced Pipeline Parallelism for DNN Training

Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators

SMSG: Profiling-Free Parallelism Modeling for Distributed Training of DNN

Accelerating Deep Neural Network guided MCTS using Adaptive Parallelism

Beyond Data and Model Parallelism for Deep Neural Networks