Abstract:Neural architecture search (NAS) has been a popular research topic for designing deep neural networks (DNNs) automatically. It is able to improve the design efficiency of neural architectures significantly for given learning tasks. Recently, instead of conducting architecture search in the original neural architecture space, many NAS approaches have been proposed to learn continuous representations from neural architectures for architecture search or estimation. In particular, Neural Architecture Optimization (NAO) is a representative method which encodes neural architectures as continuous representations by an auto-encoder and then performs continuous optimization in the encoded space with gradient-based methods. However, as NAO only considers the top-ranked architectures in learning the continuous representation, it could fail to construct a satisfied continuous optimization space which contains the expected high-quality neural architectures. Taking this cue, in this paper we propose a two-stage NAO (TNAO) to learn a more completed continuous representation of neural architectures which could provide a better optimization space for NAS. Specifically, by designing a pipeline that separates the training and search stages, we first build the training set via random sampling from the entire neural architecture search space, which is with the aim of collecting the well-distributed neural architectures for training. Moreover, to exploit the architectural semantic information with limited data effectively, we propose an improved Transformer auto-encoder for learning the continuous representation, which is supervised by ranking information of the neural architecture performance. Lastly, towards more effective optimization of neural architectures, we adopt a population-based swarm intelligence algorithm, i.e, competitive swarm optimization (CSO), with a newly designed remapping scoring scheme. To evaluate the efficiency of the proposed TNAO, comprehensive experimental studies are conducted on two common search spaces, i.e., NAS-Bench-101 and NAS-Bench-201. The architecture with the top 0.02% performance is discovered on NAS-Bench-101 and the best architecture in the CIFAR-10 dataset is obtained on NAS-Bench-201.

Neural Architecture Search for Class-incremental Learning

Efficient Architecture Search for Continual Learning

Incremental Learning with Differentiable Architecture and Forgetting Search

Exploring the Intersection Between Neural Architecture Search and Continual Learning

Continual and Multi-Task Architecture Search

Revisiting Neural Networks for Continual Learning: An Architectural Perspective

Advances in neural architecture search

Efficient Architecture Search by Network Transformation

Harnessing Neural Unit Dynamics for Effective and Scalable Class-Incremental Learning

Reinforced Continual Learning

A Study of the Learning Progress in Neural Architecture Search Techniques

Neural Architecture Search with Progressive Evaluation and Sub-Population Preservation

Two-stage Neural Architecture Optimization with Separated Training and Search.

BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture

Neural Architecture Search in Embedding Space

BNAS: Efficient Neural Architecture Search Using Broad Scalable Architecture

Differentiable Neural Architecture Learning for Efficient Neural Network Design

One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting

Automatic Convolutional Neural Architecture Search for Image Classification Under Different Scenes.

Evolving Search Space for Neural Architecture Search

Evolutionary Neural Architecture Search Combining Multi-Branch ConvNet and Improved Transformer