Abstract:Neural architecture search (NAS) has gained significant traction in automating the design of neural networks. To reduce the time cost, differentiable architecture search (DAS) transforms the traditional paradigm of discrete candidate sampling and evaluation into that of differentiable super-net optimization and discretization. However, existing DAS methods fail to trade off between model performance and model size. They either only conduct coarse-grained operation-level search, which results in redundant model parameters, or restrictively explore fine-grained filter-level and weight-level units with pre-defined remaining ratios, suffering from excessive pruning problem. Additionally, these methods compromise search quality to save memory during the search process. To tackle these issues, we introduce multi-granularity architecture search (MGAS), a unified framework which aims to discover both effective and efficient neural networks by comprehensively yet memory-efficiently exploring the multi-granularity search space. Specifically, we improve the existing DAS methods in two aspects. First, we balance the model unit numbers at different granularity levels with adaptive pruning. We learn discretization functions specific to each granularity level to adaptively determine the unit remaining ratio according to the evolving architecture. Second, we reduce the memory consumption without degrading the search quality using multi-stage search. We break down the super-net optimization and discretization into multiple sub-net stages, and perform progressive re-evaluation to allow for re-pruning and regrowing of previous units during subsequent stages, compensating for potential bias. Extensive experiments on CIFAR-10, CIFAR-100 and ImageNet demonstrate that MGAS outperforms other state-of-the-art methods in achieving a better trade-off between model performance and model size.

Milenas: Efficient neural architecture search via mixed-level reformulation

MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation

Neighborhood-Aware Neural Architecture Search

Prioritized Architecture Sampling with Monto-Carlo Tree Search

Efficient Neural Architecture Search Via Proximal Iterations

Enhanced Gradient for Differentiable Architecture Search

Understanding Architectures Learnt by Cell-based Neural Architecture Search

MGAS: Multi-Granularity Architecture Search for Trade-Off Between Model Effectiveness and Efficiency

MergeNAS: Merge Operations into One for Differentiable Architecture Search.

BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture

Generative Adversarial Neural Architecture Search

Reinforced Evolutionary Neural Architecture Search

Towards Less Constrained Macro-Neural Architecture Search

DrNAS: Dirichlet Neural Architecture Search

Making Differentiable Architecture Search less local

NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension

Multi-Objective Neural Architecture Search by Learning Search Space Partitions

DLW-NAS: Differentiable Light-Weight Neural Architecture Search

Differentiable Neural Architecture Learning for Efficient Neural Network Design

Toward Less Constrained Macro-Neural Architecture Search

Differentiable Neural Architecture Search Via Proximal Iterations.