Abstract:Automation of neural architecture design has been a coveted alternative to human experts. Various search methods have been proposed aiming to find the optimal architecture in the search space. One would expect the search results to improve when the search space grows larger since it would potentially contain more performant candidates. Surprisingly, we observe that enlarging search space is unbeneficial or even detrimental to existing NAS methods such as DARTS, ProxylessNAS, and SPOS. This counterintuitive phenomenon suggests that enabling existing methods to large search space regimes is non-trivial. However, this problem is less discussed in the literature.We present a Neural Search-space Evolution (NSE) scheme, the first neural architecture search scheme designed especially for large space neural architecture search problems. The necessity of a well-designed search space with constrained size is a tacit consent in existing methods, and our NSE aims at minimizing such necessity. Specifically, the NSE starts with a search space subset, then evolves the search space by repeating two steps: 1) search an optimized space from the search space subset, 2) refill this subset from a large pool of operations that are not traversed. We further extend the flexibility of obtainable architectures by introducing a learnable multi-branch setting. With the proposed method, we achieve 77.3% top-1 retrain accuracy on ImageNet with 333M FLOPs, which yielded a state-of-the-art performance among previous auto-generated architectures that do not involve knowledge distillation or weight pruning. When the latency constraint is adopted, our result also performs better than the previous best-performing mobile models with a 77.9% Top-1 retrain accuracy. Code is available at https://github.com/orashi/NSENAS.

A Survey on Neural Architecture Search

A Survey on Neural Architecture Search Based on Reinforcement Learning

A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions

Neural Architecture Search: Insights from 1000 Papers

Neural Architecture Search Survey: A Computer Vision Perspective

Advances in neural architecture search

A Survey on Evolutionary Neural Architecture Search

A review of neural architecture search

An Approach for Efficient Neural Architecture Search Space Definition

A Survey of Advances in Evolutionary Neural Architecture Search

Evolving Search Space for Neural Architecture Search

A Technical View on Neural Architecture Search

Trends in Neural Architecture Search: Towards the Acceleration of Search

Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search

An Introduction to Neural Architecture Search for Convolutional Networks

Automating Neural Architecture Design without Search

Evolution and Efficiency in Neural Architecture Search: Bridging the Gap Between Expert Design and Automated Optimization

Survey on Neural Network Architectures with Deep Learning

A Study of the Learning Progress in Neural Architecture Search Techniques

Task-Aware Neural Architecture Search

Understanding Neural Architecture Search Techniques