Entropic Score metric: Decoupling Topology and Size in Training-free NAS

Niccolò Cavagnero,Luca Robbiano,Francesca Pistilli,Barbara Caputo,Giuseppe Averta
2023-10-06
Abstract:Neural Networks design is a complex and often daunting task, particularly for resource-constrained scenarios typical of mobile-sized models. Neural Architecture Search is a promising approach to automate this process, but existing competitive methods require large training time and computational resources to generate accurate models. To overcome these limits, this paper contributes with: i) a novel training-free metric, named Entropic Score, to estimate model expressivity through the aggregated element-wise entropy of its activations; ii) a cyclic search algorithm to separately yet synergistically search model size and topology. Entropic Score shows remarkable ability in searching for the topology of the network, and a proper combination with LogSynflow, to search for model size, yields superior capability to completely design high-performance Hybrid Transformers for edge applications in less than 1 GPU hour, resulting in the fastest and most accurate NAS method for ImageNet classification.
Computer Vision and Pattern Recognition,Machine Learning
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to efficiently design neural network architectures in resource - constrained scenarios, especially small - scale models on mobile devices. Traditional manual design methods are challenging in finding the balance between model accuracy and computational efficiency, especially more difficult for the Transformer architecture whose computational complexity increases quadratically with the amount of input data. Although Neural Architecture Search (NAS) can automate this process, existing competitive methods require a large amount of training time and computational resources to generate accurate models, which limits the NAS methods in practical applications. To overcome these limitations, this paper makes the following contributions: 1. **Propose a new training - free metric**: Entropic Score, which is used to estimate the expressive power of a model by aggregating the element - wise entropy of activation layers. 2. **Propose a cyclic search algorithm**: This algorithm can search for model size and topology independently but cooperatively. 3. **Demonstrate the remarkable ability of Entropic Score in searching for network topologies**, and by using it in combination with LogSynflow, more efficiently fully design high - performance Hybrid Transformers suitable for edge applications. The entire search process is completed in less than 1 GPU - hour, thus becoming the fastest and most accurate NAS method in the ImageNet classification task. Specifically, Entropic Score evaluates the expressive power of a model by calculating the aggregated element - wise entropy of the output of network activation layers, and is especially suitable for the design of network topologies. LogSynflow is used to search for model size. By applying these two metrics to different aspects of network design respectively, this method can discover high - performance neural networks without training, which is not only superior to manually - designed architectures in accuracy, but also far exceeds training - based NAS methods in search efficiency.