Graph is all you need? Lightweight data-agnostic neural architecture search without training

Zhenhan Huang,Tejaswini Pedapati,Pin-Yu Chen,Chunhen Jiang,Jianxi Gao

2024-05-02

Abstract:Neural architecture search (NAS) enables the automatic design of neural network models. However, training the candidates generated by the search algorithm for performance evaluation incurs considerable computational overhead. Our method, dubbed nasgraph, remarkably reduces the computational costs by converting neural architectures to graphs and using the average degree, a graph measure, as the proxy in lieu of the evaluation metric. Our training-free NAS method is data-agnostic and light-weight. It can find the best architecture among 200 randomly sampled architectures from NAS-Bench201 in 217 CPU seconds. Besides, our method is able to achieve competitive performance on various datasets including NASBench-101, NASBench-201, and NDS search spaces. We also demonstrate that nasgraph generalizes to more challenging tasks on Micro TransNAS-Bench-101.

Machine Learning

What problem does this paper attempt to address?

The paper proposes a new method called NASGraph, aimed at addressing a problem in Neural Architecture Search (NAS): the significant computational overhead generated when evaluating the performance of candidate architectures. Traditional NAS techniques require training a large number of candidate architectures to evaluate their performance, a process that is very time-consuming and computationally expensive. To solve this problem, NASGraph adopts a training-free approach by converting neural network architectures into graph form and using the average degree of the graph as a proxy metric for performance evaluation, significantly reducing computational costs.

Specifically, the main contributions of NASGraph include:

1. Proposing a new method to map neural network architectures onto graphs. Through this mapping, the neural architecture space can be transformed into graph space. To the best of the authors' knowledge, this is the first time graph theory has been applied to neural architecture search.

2. Using the extracted graph metrics to rank neural architectures, NASGraph achieves competitive performance compared to existing training-free NAS methods in multiple benchmarks (such as NAS-Bench-101, NAS-Bench-201, Micro TransNAS-Bench-101, and NDS). Additionally, analysis shows that NASGraph has the lowest preference for operation selection, making it fairer compared to other methods.

3. Compared to existing training-free NAS techniques, NASGraph is more computationally lightweight and can be computed using only a CPU.

Through the above methods, NASGraph not only reduces the computational burden but also maintains effective prediction of neural architecture performance, making it suitable for efficient architecture search. Furthermore, the study demonstrates that NASGraph's graph metrics can be combined with existing data-dependent metrics to further improve NAS performance.