Dropout Topology-Assisted Bidirectional Learning Particle Swarm Optimization for Neural Architecture Search.

Ye-Qun Wang,Chun-Hua Chen,Jun Zhang,Zhi-Hui Zhan

DOI: https://doi.org/10.1145/3520304.3528919

2022-01-01

Abstract:The neural architecture search (NAS) is a new high-complexity optimization problem emerging in recent years. Solving NAS is challenging for optimization algorithms due to the following two issues. Firstly, besides the network architectures, there are also network hyperparameters that need to be optimized, which causes the search space of NAS to be complex and large and poses a great challenge to the optimization ability of the optimization algorithm. Secondly, NAS is an expensive optimization problem with expensive computational time to evaluate candidates, which poses a great challenge to the search speed and convergence of the optimization algorithm. Therefore, this paper proposes a novel dropout topology-assisted bidirectional learning particle swarm optimization (DBLPSO) algorithm for NAS to tackle these two issues. Firstly, inspired by the dropout technique in deep learning, a sorting-assisted dropout-based neighbor topology is proposed to enhance the population diversity and the optimization ability of PSO. Secondly, a bidirectional learning strategy is proposed to improve search speed and accelerate PSO convergence. In the experiments, the performance of the DBLPSO algorithm is evaluated on 10 tabular benchmarks based on NAS-bench 201, NATS-bench, and HPO-bench, by comparing with four NAS algorithms that have achieved state-of-the-art results on the NAS tabular benchmarks. The experimental results show that DBLPSO can obtain great performance for NAS and is superior to those NAS algorithms in comparison.