Automated Dominative Subspace Mining for Efficient Neural Architecture Search

Yaofo Chen,Yong Guo,Daihai Liao,Fanbing Lv,Hengjie Song,James Tin-Yau Kwok,Mingkui Tan
DOI: https://doi.org/10.1109/TCSVT.2024.3395463
2024-06-06
Abstract:Neural Architecture Search (NAS) aims to automatically find effective architectures within a predefined search space. However, the search space is often extremely large. As a result, directly searching in such a large search space is non-trivial and also very time-consuming. To address the above issues, in each search step, we seek to limit the search space to a small but effective subspace to boost both the search performance and search efficiency. To this end, we propose a novel Neural Architecture Search method via Dominative Subspace Mining (DSM-NAS) that finds promising architectures in automatically mined subspaces. Specifically, we first perform a global search, i.e ., dominative subspace mining, to find a good subspace from a set of candidates. Then, we perform a local search within the mined subspace to find effective architectures. More critically, we further boost search performance by taking well-designed/ searched architectures to initialize candidate subspaces. Experimental results demonstrate that DSM-NAS not only reduces the search cost but also discovers better architectures than state-of-the-art methods in various benchmark search spaces.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
### The Problem This Paper Attempts to Solve This paper aims to address two main issues in Neural Architecture Search (NAS): 1. **Large Search Space**: Traditional NAS methods typically search for effective network architectures within a predefined large search space, which is often extremely vast (e.g., approximately 10^12 architectures in ENAS, about 10^19 architectures in OFA). Directly searching within such a large space is not only very difficult but also time-consuming. 2. **Low Search Efficiency**: Exploring a large search space requires a significant amount of computational resources, which means that in practical operations, only a small portion of the architectures can be accessed. Therefore, with limited computational resources, it is challenging to obtain sufficient information to guide the architecture search. To address these issues, the paper proposes a new method called **Dominative Subspace Mining for Neural Architecture Search (DSM-NAS)**. The core idea of this method is to identify a small and effective subspace (dominative subspace) from the entire search space at each search step, thereby improving search performance and efficiency. Specifically, this goal is achieved through a two-step search process: - **Global Search**: Automatically mine a dominative subspace from a set of candidate subspaces. - **Local Search**: Further search for effective architectures within the mined subspace. Through this method, the authors hope to discover better architectures than existing methods while reducing search costs, and they have validated its effectiveness in multiple benchmark test spaces.