Abstract:Automated neural network design has received ever-increasing attention with the evolution of deep convolutional neural networks (CNNs), especially involving their deployment on embedded and mobile platforms. One of the biggest problems that neural architecture search (NAS) confronts is that a large number of candidate neural architectures are required to train, using, for instance, reinforcement learning and evolutionary optimisation algorithms, at a vast computation cost. Even recent differentiable neural architecture search (DNAS) samples a small number of candidate neural architectures based on the probability distribution of learned architecture parameters to select the final neural architecture. To address this computational complexity issue, we introduce a novel \emph{architecture parameterisation} based on scaled sigmoid function, and propose a general \emph{Differentiable Neural Architecture Learning} (DNAL) method to optimize the neural architecture without the need to evaluate candidate neural networks. Specifically, for stochastic supernets as well as conventional CNNs, we build a new channel-wise module layer with the architecture components controlled by a scaled sigmoid function. We train these neural network models from scratch. The network optimization is decoupled into the weight optimization and the architecture optimization. We address the non-convex optimization problem of neural architecture by the continuous scaled sigmoid method with convergence guarantees. Extensive experiments demonstrate our DNAL method delivers superior performance in terms of neural architecture search cost. The optimal networks learned by DNAL surpass those produced by the state-of-the-art methods on the benchmark CIFAR-10 and ImageNet-1K dataset in accuracy, model size and computational complexity.

Neural Architecture Search via Proxy Validation

Proxy Data Generation for Fast and Efficient Neural Architecture Search

Data Proxy Generation for Fast and Efficient Neural Architecture Search

EcoNAS: Finding Proxies for Economical Neural Architecture Search

Accelerating Neural Architecture Search via Proxy Data

Generative Adversarial Neural Architecture Search

Efficient Sampling for Predictor-Based Neural Architecture Search

Efficient Neural Architecture Search Via Proximal Iterations

Advances in neural architecture search

Extensible Proxy for Efficient NAS

Posterior-Guided Neural Architecture Search

Differentiable Graph Optimization for Neural Architecture Search

Automating Neural Architecture Design without Search

ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies

TG-NAS: Leveraging Zero-Cost Proxies with Transformer and Graph Convolution Networks for Efficient Neural Architecture Search

Neighborhood-Aware Neural Architecture Search

Efficient Novelty-Driven Neural Architecture Search

Neural Architecture Search in Embedding Space

Differentiable Neural Architecture Learning for Efficient Neural Network Design

A Semi-Supervised Assessor of Neural Architectures

One-Shot Neural Architecture Search Through A Posteriori Distribution Guided Sampling.