Abstract:Deep Neural Networks (DNNs) have been traditionally designed by human experts in a painstaking and expensive process, dubbed by many researchers to be more of an art than science. However, the everincreasing demand for state-of-the-art performance and real-world deployment has resulted in larger models, making the manual DNN design a daunting task. AutoML presents a promising path towards alleviating this engineering burden by automatically identifying the DNN hyperparameters, such as the number of layers or the type of layer-wise operations. As modern DNNs grow larger, AutoML methods face two key challenges: first, the increased DNN model sizes result in increased computational complexity during inference, making it difficult to deploy AutoML-designed DNNs to resource-constrained devices. Second, due to the large DNN design space, each AutoML search remains considerably costly, with an overall cost of hundreds of GPU-hours. In this thesis, we propose AutoML methods that are both hardware aware and search-cost efficient. We introduce a Bayesian optimization (BO) methodology enhanced with hardware-cost predictive models, allowing the AutoML search to traverse the design space in a constraint “complying” manner, up to 3.5× faster compared to vanilla BO methods. Moreover, we formulate the design of adaptive DNNs as an AutoML task and we jointly solve for the DNN architectures and the adaptive execution scheme, reducing energy consumption by up to 6× compared to hand-tuned designs. Next, in a departure from existing one-shot Neural Architecture Search (NAS) assumptions on how the candidate DNN architectures are evaluated, we introduce a novel view of the one-shot NAS problem as finding the subsets of kernel weights across a single-path one-shot model. Our proposed formulation reduces the NAS search cost by up to 5,000× compared to existing NAS methods. Taking advantage of such efficiency, we investigate how various design space and formulation choices affect the AutoML results, achieving a new state-of-the-art NAS performance for image classification accuracy (75.62%) under runtime constraints on mobile devices.

Implication of Optimizing NPU Dataflows on Neural Architecture Search for Mobile Devices

DPP-Net: Device-Aware Progressive Search for Pareto-Optimal Neural Architectures

Profiling Neural Blocks and Design Spaces for Mobile Neural Architecture Search

One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search

Single-Path NAS: Designing Hardware-Efficient ConvNets in Less Than 4 Hours

S3NAS: Fast NPU-aware Neural Architecture Search Methodology

Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization

Latency-aware Neural Architecture Performance Predictor with Query-to-Tier Technique

Single-Path NAS: Device-Aware Efficient ConvNet Design

MnasFPN: Learning Latency-Aware Pyramid Architecture for Object Detection on Mobile Devices

U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search

Hardware-Aware AutoML for Efficient Deep Learning Applications

FastDeepIoT: Towards Understanding and Optimizing Neural Network Execution Time on Mobile and Embedded Devices

DLW-NAS: Differentiable Light-Weight Neural Architecture Search

NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization

NASH: Neural Architecture Search for Hardware-Optimized Machine Learning Models

MobileDets: Searching for Object Detection Architectures for Mobile Accelerators

Search-time Efficient Device Constraints-Aware Neural Architecture Search

Accuracy vs. Efficiency: Achieving Both through FPGA-Implementation Aware Neural Architecture Search

Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces

Generalized Latency Performance Estimation for Once-For-All Neural Architecture Search