Abstract:The differentiable neural architecture search (NAS) framework has obtained extensive attention and achieved remarkable performance due to its search efficiency. However, most existing differentiable NAS methods still suffer from issues of model collapse, degenerated search-evaluation correlation, and inefficient hardware deployment, which causes the searched architectures to be suboptimal in accuracy and cannot meet different computation resource constraints (e.g., FLOPs and latency). In this article, we propose a novel resource-constrained NAS (ReCNAS) method, which can efficiently search high-performance architectures that satisfy the given constraints, and deal with the issues observed in previous differentiable NAS methods from three aspects: search space, search strategy, and resource adaptability. First, we introduce an elastic densely connected layerwise search space, which decouples the architecture depth representation from the search of candidate operations to alleviate the aggregation of skip connections and architecture redundancies. Second, a scheme of group annealing and progressive pruning is proposed to improve the efficiency and bridge the search-evaluation gap, which steadily forces the architecture parameters close to binary distribution and progressively prunes the inferior operations. Third, we present a novel resource-constrained architecture generation method, which prunes the redundant channel throughout the search based on dynamic programming, making the searched architecture scalable to different devices and requirements. Extensive experimental results demonstrate the efficiency and search stability of our ReCNAS, which is capable of discovering high-performance architectures on different datasets and tasks, surpassing other NAS methods, while tightly meeting the target resource constraints without any tuning required. Besides, the searched architectures show strong generalizability to other complex vision tasks.

Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces

DDPNAS: Efficient Neural Architecture Search Via Dynamic Distribution Pruning

Multinomial Distribution Learning for Effective Neural Architecture Search

Efficient Architecture Search for Diverse Tasks

Evolving Search Space for Neural Architecture Search

RelativeNAS: Relative Neural Architecture Search Via Slow-Fast Learning

NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization

You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization

Neighborhood-Aware Neural Architecture Search

Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective

AutoSpace: Neural Architecture Search with Less Human Interference

MNGNAS: Distilling Adaptive Combination of Multiple Searched Networks for One-Shot Neural Architecture Search

Multi-Objective Neural Architecture Search Based on Diverse Structures and Adaptive Recommendation

Data Aware Neural Architecture Search

Searching for A Robust Neural Architecture in Four GPU Hours

HSCoNAS: Hardware-Software Co-Design of Efficient DNNs Via Neural Architecture Search

ReCNAS: Resource-Constrained Neural Architecture Search Based on Differentiable Annealing and Dynamic Pruning.

Search-time Efficient Device Constraints-Aware Neural Architecture Search

Differentiable Neural Architecture Learning for Efficient Neural Network Design

Block-wisely Supervised Neural Architecture Search with Knowledge Distillation

Fine-Grained Stochastic Architecture Search