Abstract:Most conventional Neural Architecture Search (NAS) approaches are limited in that they only generate architectures without searching for the optimal parameters. While some NAS methods handle this issue by utilizing a supernet trained on a large-scale dataset such as ImageNet, they may be suboptimal if the target tasks are highly dissimilar from the dataset the supernet is trained on. To address such limitations, we introduce a novel problem of \emph{Neural Network Search} (NNS), whose goal is to search for the optimal pretrained network for a novel dataset and constraints (e.g. number of parameters), from a model zoo. Then, we propose a novel framework to tackle the problem, namely \emph{Task-Adaptive Neural Network Search} (TANS). Given a model-zoo that consists of network pretrained on diverse datasets, we use a novel amortized meta-learning framework to learn a cross-modal latent space with contrastive loss, to maximize the similarity between a dataset and a high-performing network on it, and minimize the similarity between irrelevant dataset-network pairs. We validate the effectiveness and efficiency of our method on ten real-world datasets, against existing NAS/AutoML baselines. The results show that our method instantly retrieves networks that outperform models obtained with the baselines with significantly fewer training steps to reach the target performance, thus minimizing the total cost of obtaining a task-optimal network. Our code and the model-zoo are available at <a class="link-external link-https" href="https://github.com/wyjeong/TANS" rel="external noopener nofollow">this https URL</a>.

Task-Aware Neural Architecture Search

Mitigating Search Interference With Task-Aware Nested Search

Toward Evolutionary Multitask Convolutional Neural Architecture Search

Across-task Neural Architecture Search Via Meta Learning

Advances in neural architecture search

Automatic Subspace Evoking for Efficient Neural Architecture Search

Data Aware Neural Architecture Search

Fast Task-Aware Architecture Inference

Automating Neural Architecture Design without Search

Efficient Architecture Search for Diverse Tasks

An Approach for Efficient Neural Architecture Search Space Definition

A Technical View on Neural Architecture Search

Arch-Graph: Acyclic Architecture Relation Predictor for Task-Transferable Neural Architecture Search

Automated Dominative Subspace Mining for Efficient Neural Architecture Search

Understanding Architectures Learnt by Cell-based Neural Architecture Search

Generative Adversarial Neural Architecture Search

Neighborhood-Aware Neural Architecture Search

Efficient Gradient-Based Neural Architecture Search for End-to-End ASR

Task-Adaptive Neural Network Search with Meta-Contrastive Learning

AtomNAS: Fine-Grained End-to-End Neural Architecture Search

Manas: multi-agent neural architecture search