Abstract:Designing effective architectures is one of the key factors behind the success of deep neural networks. Existing deep architectures are either manually designed or automatically searched by some Neural Architecture Search (NAS) methods. However, even a well-designed/searched architecture may still contain many nonsignificant or redundant modules/operations (e.g., some intermediate convolution or pooling layers). Such redundancy may not only incur substantial memory consumption and computational cost but also deteriorate the performance. Thus, it is necessary to optimize the operations inside an architecture to improve the performance without introducing extra computational cost. To this end, we have proposed a Neural Architecture Transformer (NAT) method which casts the optimization problem into a Markov Decision Process (MDP) and seeks to replace the redundant operations with more efficient operations, such as skip or null connection. Note that NAT only considers a small number of possible replacements/transitions and thus comes with a limited search space. As a result, such a small search space may hamper the performance of architecture optimization. To address this issue, we propose a Neural Architecture Transformer++ (NAT++) method which further enlarges the set of candidate transitions to improve the performance of architecture optimization. Specifically, we present a two-level transition rule to obtain valid transitions, i.e., allowing operations to have more efficient types (e.g., convolution -> separable convolution) or smaller kernel sizes (e.g., 5x5 -> 3x3). Note that different operations may have different valid transitions. We further propose a Binary-Masked Softmax (BMSoftmax) layer to omit the possible invalid transitions. Last, based on the MDP formulation, we apply policy gradient to learn an optimal policy, which will be used to infer the optimized architectures. Extensive experiments show that the transformed architectures significantly outperform both their original counterparts and the architectures optimized by existing methods.

Inter-layer transition in neural architecture search

Understanding Architectures Learnt by Cell-based Neural Architecture Search

Latency-aware Neural Architecture Performance Predictor with Query-to-Tier Technique

NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension

Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift

Delta-NAS: Difference of Architecture Encoding for Predictor-based Evolutionary Neural Architecture Search

Towards Accurate and Compact Architectures Via Neural Architecture Transformer

Differentiable Neural Architecture Search in Equivalent Space with Exploration Enhancement.

Dependency-aware Differentiable Neural Architecture Search

Architecture Self-Attention Mechanism: Nonlinear Optimization for Neural Architecture Search

Discretization-aware architecture search

Understanding Neural Architecture Search Techniques

How Does Supernet Help in Neural Architecture Search?

Distribution Consistent Neural Architecture Search

A Hierarchical Evolution of Neural Architecture Search Method Based on State Transition Algorithm

Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search

NAT: Neural Architecture Transformer for Accurate and Compact Architectures

LayerNAS: Neural Architecture Search in Polynomial Complexity

Differentiable Neural Architecture Learning for Efficient Neural Network Design

Automating Neural Architecture Design without Search

Stage-Wise Neural Architecture Search