Abstract:Neural Architecture Search (NAS) is a powerful tool for automating effective image processing DNN designing. The ranking has been advocated to design an efficient performance predictor for NAS. The previous contrastive method solves the ranking problem by comparing pairs of architectures and predicting their relative performance. However, it only focuses on the rankings between two involved architectures and neglects the overall quality distributions of the search space, which may suffer generalization issues. A predictor, namely Neural Architecture Ranker (NAR) which concentrates on the global quality tier of specific architecture, is proposed to tackle such problems caused by the local perspective. The NAR explores the quality tiers of the search space globally and classifies each individual to the tier they belong to according to its global ranking. Thus, the predictor gains the knowledge of the performance distributions of the search space which helps to generalize its ranking ability to the datasets more easily. Meanwhile, the global quality distribution facilitates the search phase by directly sampling candidates according to the statistics of quality tiers, which is free of training a search algorithm, e.g., Reinforcement Learning (RL) or Evolutionary Algorithm (EA), thus it simplifies the NAS pipeline and saves the computational overheads. The proposed NAR achieves better performance than the state-of-the-art methods on two widely used datasets for NAS research. On the vast search space of NAS-Bench-101, the NAR easily finds the architecture with top 0.01 performance only by sampling. It also generalizes well to different image datasets of NAS-Bench-201, i.e., CIFAR-10, CIFAR-100, and ImageNet-16-120 by identifying the optimal architectures for each of them.

GLiT: Neural Architecture Search for Global and Local Image Transformer

Neural Architecture Search on Efficient Transformers and Beyond

Training-free Neural Architectural Search on Transformer Via Evaluating Expressivity and Trainability

NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict Aware Supernet Training

Generalized Global Ranking-Aware Neural Architecture Ranker for Efficient Image Classifier Search

Searching the Search Space of Vision Transformer

Neural Architecture Search with a Lightweight Transformer for Text-to-Image Synthesis

BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search

Searching Better Architectures for Neural Machine Translation

UniNet: Unified Architecture Search with Convolution, Transformer, and MLP

HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers

Local-to-Global Self-Attention in Vision Transformers

Training-free Neural Architecture Search for RNNs and Transformers

AutoFormer: Searching Transformers for Visual Recognition

Global-Local Similarity for Efficient Fine-Grained Image Recognition with Vision Transformers

Neural Architecture Search Via Combinatorial Multi-Armed Bandit.

Efficient Architecture Search by Network Transformation

EnTranNAS: Towards Closing the Gap between the Architectures in Search and Evaluation

TG-NAS: Leveraging Zero-Cost Proxies with Transformer and Graph Convolution Networks for Efficient Neural Architecture Search

Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective