Abstract:Neural architecture search (NAS) has gained significant traction in automating the design of neural networks. To reduce the time cost, differentiable architecture search (DAS) transforms the traditional paradigm of discrete candidate sampling and evaluation into that of differentiable super-net optimization and discretization. However, existing DAS methods fail to trade off between model performance and model size. They either only conduct coarse-grained operation-level search, which results in redundant model parameters, or restrictively explore fine-grained filter-level and weight-level units with pre-defined remaining ratios, suffering from excessive pruning problem. Additionally, these methods compromise search quality to save memory during the search process. To tackle these issues, we introduce multi-granularity architecture search (MGAS), a unified framework which aims to discover both effective and efficient neural networks by comprehensively yet memory-efficiently exploring the multi-granularity search space. Specifically, we improve the existing DAS methods in two aspects. First, we balance the model unit numbers at different granularity levels with adaptive pruning. We learn discretization functions specific to each granularity level to adaptively determine the unit remaining ratio according to the evolving architecture. Second, we reduce the memory consumption without degrading the search quality using multi-stage search. We break down the super-net optimization and discretization into multiple sub-net stages, and perform progressive re-evaluation to allow for re-pruning and regrowing of previous units during subsequent stages, compensating for potential bias. Extensive experiments on CIFAR-10, CIFAR-100 and ImageNet demonstrate that MGAS outperforms other state-of-the-art methods in achieving a better trade-off between model performance and model size.

MGAS: Multi-Granularity Architecture Search for Trade-Off Between Model Effectiveness and Efficiency

Fine-Grained Stochastic Architecture Search

Prioritized Architecture Sampling with Monto-Carlo Tree Search

$\alpha$ DARTS Once More: Enhancing Differentiable Architecture Search by Masked Image Modeling

Enhanced Gradient for Differentiable Architecture Search

Multi-Objective Neural Architecture Search for Light-Weight Model

SGAS: Sequential Greedy Architecture Search

MC-PDARTS: Multi-Cell Progressive Differentiable Architecture Search.

Multi-Objective Neural Architecture Search Based on Diverse Structures and Adaptive Recommendation

Enhanced Differentiable Architecture Search Based on Asymptotic Regularization

Differentiable Neural Architecture Learning for Efficient Neural Network Design

MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation

Discretization-aware architecture search

Manas: multi-agent neural architecture search

Unchain the Search Space with Hierarchical Differentiable Architecture Search

Searching for A Robust Neural Architecture in Four GPU Hours

Efficient Gradient-Based Neural Architecture Search for End-to-End ASR

Automated Search-Space Generation Neural Architecture Search

Making Differentiable Architecture Search less local

LMD-DARTS: Low-Memory, Densely Connected, Differentiable Architecture Search