Abstract:Weight-sharing supernets are crucial for performance estimation in cutting-edge neural architecture search (NAS) frameworks. Despite their ability to generate diverse subnetworks without retraining, the quality of these subnetworks is not guaranteed due to weight sharing. In NLP tasks like machine translation and pre-trained language modeling, there is a significant performance gap between supernet and training from scratch for the same model architecture, necessitating retraining post optimal architecture identification. This study introduces a solution called mixture-of-supernets, a generalized supernet formulation leveraging mixture-of-experts (MoE) to enhance supernet model expressiveness with minimal training overhead. Unlike conventional supernets, this method employs an architecture-based routing mechanism, enabling indirect sharing of model weights among subnetworks. This customization of weights for specific architectures, learned through gradient descent, minimizes retraining time, significantly enhancing training efficiency in NLP. The proposed method attains state-of-the-art (SoTA) performance in NAS for fast machine translation models, exhibiting a superior latency-BLEU tradeoff compared to HAT, the SoTA NAS framework for machine translation. Furthermore, it excels in NAS for building memory-efficient task-agnostic BERT models, surpassing NAS-BERT and AutoDistil across various model sizes. The code can be found at: <a class="link-external link-https" href="https://github.com/UBC-NLP/MoS" rel="external noopener nofollow">this https URL</a>.

NES-TL: Network Embedding Similarity-Based Transfer Learning

TranGAN: Generative Adversarial Network Based Transfer Learning for Social Tie Prediction.

Extreme Learning Machine Based Transfer Learning for Data Classification.

Deep Model Transferability from Attribution Maps

Sequential Transfer Machine Learning in Networks: Measuring the Impact of Data and Neural Net Similarity on Transferability

DA-Net: A Disentangled and Adaptive Network for Multi-Source Cross-Lingual Transfer Learning

Active Domain Transfer on Network Embedding.

Augmenting Knowledge Transfer across Graphs

Robust Metric Boosts Transfer

Transferrable Prototypical Networks for Unsupervised Domain Adaptation

Automatic Knowledge Fusion in Transferrable Networks for Semantic Text Matching.

Transferable and Differentiable Discrete Network Embedding for multi-domains with Hierarchical Knowledge Distillation

Network Grafting: Transferring Learned Features from Trained Neural Networks

Neural Embedding Propagation on Heterogeneous Networks

Exploring and Predicting Transferability across NLP Tasks

Enhancing Knowledge Transfer for Task Incremental Learning with Data-free Subnetwork.

Deep Transfer Neural Network Using Hybrid Representations of Domain Discrepancy

Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts

Heterogeneous Transfer Learning for Image Classification.

Transferable graph auto-encoders for cross-network node classification

Cross-domain NER under a Divide-and-Transfer Paradigm