Abstract:Recent progress on neural architecture search (NAS) has demonstrated exciting results on automating deep network architecture designs. In order to overcome the unaffordable complexity of training each candidate architecture from scratch, the state-of-the-art one-shot NAS approaches adopt a weight-sharing strategy to improve training efficiency. Although the computational cost is greatly reduced, such oneshot process introduces a severe weight coupling problem that largely degrades the evaluation accuracy of each candidate. The existing approaches often address the problem by shrinking the search space, model distillation, or fewshot training. Instead, in this paper, we propose a novel distribution consistent one-shot neural architecture search algorithm. We first theoretically investigate how the weight coupling problem affects the network searching performance from a parameter distribution perspective, and then propose a novel supernet training strategy with a Distribution Consistent Constraint that can provide a good measurement for the extent to which two architectures can share weights. Our strategy optimizes the supernet through iteratively inferring network weights and corresponding local sharing states. Such joint optimization of supernet's weights and topologies can diminish the discrepancy between the weights inherited from the supernet and the ones that are trained with a stand-alone model. As a result, it enables a more accurate model evaluation phase and leads to a better searching performance. We conduct extensive experiments on benchmark datasets with multiple searching spaces. The resulting architecture achieves superior performance over the current state-of-the-art NAS algorithms with comparable search costs, which demonstrates the efficacy of our approach.

Powering One-Shot Topological NAS with Stabilized Share-Parameter Proxy.

Multi-shot NAS for Discovering Adversarially Robust Convolutional Neural Architectures at Targeted Capacities

K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets

TopoNAS: Boosting Search Efficiency of Gradient-based NAS via Topological Simplification

GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet

Prioritized Architecture Sampling with Monto-Carlo Tree Search

PWSNAS: Powering Weight Sharing NAS With General Search Space Shrinking Framework

Balanced One-shot Neural Architecture Optimization

One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting

Understanding and Improving One-shot Neural Architecture Optimization

Distribution Consistent Neural Architecture Search

Improving One-Shot NAS with Shrinking-and-Expanding Supernet

Single Path One-Shot Neural Architecture Search with Uniform Sampling

One-Shot Neural Architecture Search Through A Posteriori Distribution Guided Sampling.

NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search

SiGeo: Sub-One-Shot NAS via Information Theory and Geometry of Loss Landscape

Discovering Robust Convolutional Architecture at Targeted Capacity: A Multi-Shot Approach

Zero-Shot NAS via the Suppression of Local Entropy Decrease

Overcoming Multi-Model Forgetting in One-Shot NAS With Diversity Maximization

BayesNAS: A Bayesian Approach for Neural Architecture Search

RD-NAS: Enhancing One-Shot Supernet Ranking Ability Via Ranking Distillation from Zero-Cost Proxies