Abstract:Automated neural network design has received ever-increasing attention with the evolution of deep convolutional neural networks (CNNs), especially involving their deployment on embedded and mobile platforms. One of the biggest problems that neural architecture search (NAS) confronts is that a large number of candidate neural architectures are required to train, using, for instance, reinforcement learning and evolutionary optimisation algorithms, at a vast computation cost. Even recent differentiable neural architecture search (DNAS) samples a small number of candidate neural architectures based on the probability distribution of learned architecture parameters to select the final neural architecture. To address this computational complexity issue, we introduce a novel \emph{architecture parameterisation} based on scaled sigmoid function, and propose a general \emph{Differentiable Neural Architecture Learning} (DNAL) method to optimize the neural architecture without the need to evaluate candidate neural networks. Specifically, for stochastic supernets as well as conventional CNNs, we build a new channel-wise module layer with the architecture components controlled by a scaled sigmoid function. We train these neural network models from scratch. The network optimization is decoupled into the weight optimization and the architecture optimization. We address the non-convex optimization problem of neural architecture by the continuous scaled sigmoid method with convergence guarantees. Extensive experiments demonstrate our DNAL method delivers superior performance in terms of neural architecture search cost. The optimal networks learned by DNAL surpass those produced by the state-of-the-art methods on the benchmark CIFAR-10 and ImageNet-1K dataset in accuracy, model size and computational complexity.

Differentiable Graph Optimization for Neural Architecture Search

Graph Differentiable Architecture Search with Structure Learning

Efficient Neural Architecture Search Via Proximal Iterations

Enhanced Gradient for Differentiable Architecture Search

Differentiable Neural Architecture Search Via Proximal Iterations.

Graph is all you need? Lightweight data-agnostic neural architecture search without training

Evolving graph convolutional networks for neural architecture search

DiffMG: Differentiable Meta Graph Search for Heterogeneous Graph Neural Networks

Differentiable Neural Architecture Learning for Efficient Neural Network Design

Multi-Objective Neural Architecture Search by Learning Search Space Partitions

Prioritized Architecture Sampling with Monto-Carlo Tree Search

Simplifying Architecture Search for Graph Neural Network

Probabilistic Dual Network Architecture Search on Graphs

Making Differentiable Architecture Search less local

Generative Adversarial Neural Architecture Search

Enhanced Differentiable Architecture Search Based on Asymptotic Regularization

Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search

Searching for A Robust Neural Architecture in Four GPU Hours

Evolution and Efficiency in Neural Architecture Search: Bridging the Gap Between Expert Design and Automated Optimization

Search Space Adaptation for Differentiable Neural Architecture Search in Image Classification

Neighborhood-Aware Neural Architecture Search