Abstract:Deep Neural Networks (DNNs) have been traditionally designed by human experts in a painstaking and expensive process, dubbed by many researchers to be more of an art than science. However, the everincreasing demand for state-of-the-art performance and real-world deployment has resulted in larger models, making the manual DNN design a daunting task. AutoML presents a promising path towards alleviating this engineering burden by automatically identifying the DNN hyperparameters, such as the number of layers or the type of layer-wise operations. As modern DNNs grow larger, AutoML methods face two key challenges: first, the increased DNN model sizes result in increased computational complexity during inference, making it difficult to deploy AutoML-designed DNNs to resource-constrained devices. Second, due to the large DNN design space, each AutoML search remains considerably costly, with an overall cost of hundreds of GPU-hours. In this thesis, we propose AutoML methods that are both hardware aware and search-cost efficient. We introduce a Bayesian optimization (BO) methodology enhanced with hardware-cost predictive models, allowing the AutoML search to traverse the design space in a constraint “complying” manner, up to 3.5× faster compared to vanilla BO methods. Moreover, we formulate the design of adaptive DNNs as an AutoML task and we jointly solve for the DNN architectures and the adaptive execution scheme, reducing energy consumption by up to 6× compared to hand-tuned designs. Next, in a departure from existing one-shot Neural Architecture Search (NAS) assumptions on how the candidate DNN architectures are evaluated, we introduce a novel view of the one-shot NAS problem as finding the subsets of kernel weights across a single-path one-shot model. Our proposed formulation reduces the NAS search cost by up to 5,000× compared to existing NAS methods. Taking advantage of such efficiency, we investigate how various design space and formulation choices affect the AutoML results, achieving a new state-of-the-art NAS performance for image classification accuracy (75.62%) under runtime constraints on mobile devices.

Automatic Generation of Dynamic Inference Architecture for Deep Neural Networks

S2DNAS: Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search

A Progressive Subnetwork Searching Framework for Dynamic Inference

A Progressive Sub-Network Searching Framework for Dynamic Inference

Differentiable Neural Architecture Learning for Efficient Neural Network Design

NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization

Automating Neural Architecture Design without Search

Advances in neural architecture search

Automated Search-Space Generation Neural Architecture Search

AutoSpace: Neural Architecture Search with Less Human Interference

AutoIAS

Deeply Supervised Block-Wise Neural Architecture Search

Efficient Architecture Search by Network Transformation

Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild

Hardware-Aware AutoML for Efficient Deep Learning Applications

NAT: Neural Architecture Transformer for Accurate and Compact Architectures

A Technical View on Neural Architecture Search

NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension

Efficient Architecture Search via Bi-level Data Pruning

DrNAS: Dirichlet Neural Architecture Search

Differentiable neural architecture learning for efficient neural networks