Abstract:One-shot neural architecture search (NAS) has recently become mainstream in the NAS community because it significantly improves computational efficiency through weight sharing. However, the supernet training paradigm in one-shot NAS introduces catastrophic forgetting, where each step of the training can deteriorate the performance of other architectures that contain partially-shared weights with current architecture. To overcome this problem of catastrophic forgetting, we formulate supernet training for one-shot NAS as a constrained continual learning optimization problem such that learning the current architecture does not degrade the validation accuracy of previous architectures. The key to solving this constrained optimization problem is a novelty search based architecture selection (NSAS) loss function that regularizes the supernet training by using a greedy novelty search method to find the most representative subset. We applied the NSAS loss function to two one-shot NAS baselines and extensively tested them on both a common search space and a NAS benchmark dataset. We further derive three variants based on the NSAS loss function, the NSAS with depth constrain (NSAS-C) to improve the transferability, and NSAS-G and NSAS-LG to handle the situation with a limited number of constraints. The experiments on the common NAS search space demonstrate that NSAS and it variants improve the predictive ability of supernet training in one-shot NAS with remarkable and efficient performance on the CIFAR-10, CIFAR-100, and ImageNet datasets. The results with the NAS benchmark dataset also confirm the significant improvements these one-shot NAS baselines can make.

Rethinking Bi-Level Optimization in Neural Architecture Search: A Gibbs Sampling Perspective.

One-Shot Neural Architecture Search Through A Posteriori Distribution Guided Sampling.

Prioritized Architecture Sampling with Monto-Carlo Tree Search

Posterior-Guided Neural Architecture Search

DATA: Differentiable ArchiTecture Approximation with Distribution Guided Sampling

Neural Architecture Selection as a Nash Equilibrium With Batch Entanglement

BayesNAS: A Bayesian Approach for Neural Architecture Search

DrNAS: Dirichlet Neural Architecture Search

SNAS: Stochastic Neural Architecture Search

Single-DARTS: Towards Stable Architecture Search

Making Differentiable Architecture Search less local

One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting

Differentiable Neural Architecture Search Via Proximal Iterations.

Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild

NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search

Neighborhood-Aware Neural Architecture Search

ZARTS: on Zero-order Optimization for Neural Architecture Search

Single Path One-Shot Neural Architecture Search with Uniform Sampling

TopoNAS: Boosting Search Efficiency of Gradient-based NAS via Topological Simplification

Neural Ensemble Search via Bayesian Sampling

Balanced One-shot Neural Architecture Optimization