Abstract:Neural architecture search (NAS) is an effective approach for automating the design of deep neural networks. Evolutionary computation (EC) is commonly used in NAS due to its global optimization capability. However, the evaluation phase of architecture candidates in EC-based NAS is compute-intensive, limiting its application for many real-world problems. To overcome this challenge, we propose a novel progressive evaluation strategy for the evaluation phase in convolutional neural network architecture search, in which the number of training epochs of network individuals is progressively increased. In addition, a sub-population preservation strategy is proposed to preserve medium-size and large-size architectures to avoid prematurely discarding networks that may not perform well in the early stages but have the potential to excel with further optimization. Our proposed algorithm reduces the computational cost of the evaluation phase and promotes population diversity and fairness by preserving promising networks based on their distribution. We evaluate the proposed progressive evaluation and sub-population preservation of neural architecture search (PEPNAS) algorithm on the CIFAR10, CIFAR100, and ImageNet benchmark datasets, and compare it with 36 state-of-the-art algorithms, including manually designed networks, reinforcement learning (RL) algorithms, gradient-based algorithms, and other EC-based ones. The experimental results demonstrate that PEPNAS effectively identifies networks with competitive accuracy while also markedly improving the efficiency of the search process. For instance, PEPNAS discovers the architecture on CIFAR10 with a low error rate of 2.38% using only 0.7 GPU days. We directly adopt the searched architecture for the image classification on the CIFAR100 and ImageNet datasets, which achieves the top 1 error rates of 16.46% and 26.25%, respectively. The code is available at https://github.com/chajiajie/PEPNAS.

Action Command Encoding for Surrogate-Assisted Neural Architecture Search

A Study on Encodings for Neural Architecture Search

Similarity surrogate-assisted evolutionary neural architecture search with dual encoding strategy

A Generic Graph-Based Neural Architecture Encoding Scheme with Multifaceted Information

Encodings for Prediction-based Neural Architecture Search

Generalized Global Ranking-Aware Neural Architecture Ranker for Efficient Image Classifier Search

TA-GATES: An Encoding Scheme for Neural Network Architectures

Ranking‐based architecture generation for surrogate‐assisted neural architecture search

Neural Architecture Search in Embedding Space

Latency-aware Neural Architecture Performance Predictor with Query-to-Tier Technique

Graph-based Neural Architecture Search with Operation Embeddings

Understanding Architectures Learnt by Cell-based Neural Architecture Search

Neural Architecture Search with Progressive Evaluation and Sub-Population Preservation

Delta-NAS: Difference of Architecture Encoding for Predictor-based Evolutionary Neural Architecture Search

A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS

A Study of the Learning Progress in Neural Architecture Search Techniques

A Survey on Computationally Efficient Neural Architecture Search

einspace: Searching for Neural Architectures from Fundamental Operations

Neural Architecture Search on Efficient Transformers and Beyond

Guided evolutionary Neural Architecture Search with efficient performance estimation

NAAS: Neural Accelerator Architecture Search