Abstract:Neural architecture search (NAS) depends heavily on an efficient and accurate performance estimator. To speed up the evaluation process, recent advances, like differentiable architecture search (DARTS) and One-Shot approaches, instead of training every model from scratch, train a weight-sharing super-network to reuse parameters among different candidates, in which all child models can be efficiently evaluated. Though these methods significantly boost search efficiency, they inherently suffer from inaccurate and unstable performance estimation. To this end, we propose a general and effective framework for powering weight-sharing NAS, namely, PWSNAS, by shrinking search space automatically, i.e., candidate operators will be discarded if they are less important. With the strategy, our approach can provide a promising search space of a smaller size by progressively simplifying the original search space, which can reduce difficulties for existing NAS methods to find superior architectures. In particular, we present two strategies to guide the shrinking process: detect redundant operators with a new angle-based metric and decrease the degree of weight sharing of a super-network by increasing parameters, which differentiates PWSNAS from existing shrinking methods. Comprehensive analysis experiments on NASBench-201 verify the superiority of our proposed metric over existing accuracy-based and magnitude-based metrics. PWSNAS can easily apply to the state-of-the-art NAS methods, e.g., single path one-shot neural architecture search (SPOS), FairNAS, ProxylessNAS, DARTS, and progressive DARTS (PDARTS). We evaluate PWSNAS and demonstrate consistent performance gains over baseline methods.

Improving One-Shot NAS with Shrinking-and-Expanding Supernet

K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets

GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet

PWSNAS: Powering Weight Sharing NAS With General Search Space Shrinking Framework

One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting

GreedyNASv2: Greedier Search with a Greedy Path Filter

CLOSE: Curriculum Learning on the Sharing Extent Towards Better One-Shot NAS

SCARLET-NAS: Bridging the Gap between Stability and Scalability in Weight-sharing Neural Architecture Search

Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach

Multi-shot NAS for Discovering Adversarially Robust Convolutional Neural Architectures at Targeted Capacities

Efficient Novelty-Driven Neural Architecture Search

PHD-NAS: Preserving helpful data to promote Neural Architecture Search

SiGeo: Sub-One-Shot NAS via Information Theory and Geometry of Loss Landscape

One-Shot Neural Architecture Search by Dynamically Pruning Supernet in Hierarchical Order

How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS

MNGNAS: Distilling Adaptive Combination of Multiple Searched Networks for One-Shot Neural Architecture Search

Prioritized Architecture Sampling with Monto-Carlo Tree Search

NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension

AlphaNet: Improved Training of Supernets with Alpha-Divergence

Deeper Insights into Weight Sharing in Neural Architecture Search

Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts