Abstract:As the ubiquity of deep learning in various machine learning applications has amplified, a proliferation of neural network models has been trained and shared on public model repositories. In the context of a targeted machine learning assignment, utilizing an apt source model as a starting point typically outperforms the strategy of training from scratch, particularly with limited training data. Despite the investigation and development of numerous model selection strategies in prior work, the process remains time-consuming, especially given the ever-increasing scale of model repositories. In this paper, we propose a two-phase (coarse-recall and fine-selection) model selection framework, aiming to enhance the efficiency of selecting a robust model by leveraging the models' training performances on benchmark datasets. Specifically, the coarse-recall phase clusters models showcasing similar training performances on benchmark datasets in an offline manner. A light-weight proxy score is subsequently computed between this model cluster and the target dataset, which serves to recall a significantly smaller subset of potential candidate models in a swift manner. In the following fine-selection phase, the final model is chosen by fine-tuning the recalled models on the target dataset with successive halving. To accelerate the process, the final fine-tuning performance of each potential model is predicted by mining the model's convergence trend on the benchmark datasets, which aids in filtering lower performance models more earlier during fine-tuning. Through extensive experimentation on tasks covering natural language processing and computer vision, it has been demonstrated that the proposed methodology facilitates the selection of a high-performing model at a rate about 3x times faster than conventional baseline methods. Our code is available at

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to efficiently select pre - trained models suitable for specific machine - learning tasks in large - scale model libraries. With the popularization of deep learning in various machine - learning applications, a large number of neural network models have been trained and shared in public model libraries. For a specific machine - learning task, using a suitable source model as a starting point usually performs better than training from scratch, especially when the training data is limited. However, although previous research has developed a variety of model selection strategies, as the scale of the model library continues to grow, the selection process is still time - consuming. The paper proposes a two - stage (coarse recall and fine selection) model selection framework, aiming to improve the efficiency of selecting robust models by leveraging the training performance of models on benchmark datasets. Specifically, the coarse recall in the first stage clusters models showing similar training performance in an offline manner and calculates lightweight proxy scores between these model clusters and the target dataset, thereby quickly recalling a small number of potential candidate models. The fine selection in the second stage fine - tunes the recalled models on the target dataset and uses the successive halving method to select the final model. To accelerate this process, the final fine - tuning performance of each potential model is also predicted by mining the convergence trend of the model on the benchmark dataset, which helps to filter out models with lower performance in the early stage of fine - tuning. Verified through extensive experiments covering natural language processing and computer vision tasks, the proposed method can select high - performance models at approximately three times the speed of traditional baseline methods.

A Two-Phase Recall-and-Select Framework for Fast Model Selection

Towards Fundamentally Scalable Model Selection: Asymptotically Fast Update and Selection

Selection via Proxy: Efficient Data Selection for Deep Learning

A CLIP-Powered Framework for Robust and Generalizable Data Selection

A Fast Automated Model Selection Approach Based on Collaborative Knowledge

Budgeted Online Model Selection and Fine-Tuning via Federated Learning

Towards Accelerated Model Training via Bayesian Data Selection

Knockoff-Guided Feature Selection via A Single Pre-trained Reinforced Agent

Palette: Towards Multi-Source Model Selection And Ensemble For Reuse

An Efficient Source Model Selection Framework in Model Databases

Exploring Dense Retrieval for Dialogue Response Selection

Deep Neural Network Benchmarks for Selective Classification

An Approach With Low Redundancy To Network Feature Selection Based On Multiple Order Proximity

Selective Annotation Makes Language Models Better Few-Shot Learners

Robust Image Matching By Dynamic Feature Selection

Diversified Batch Selection for Training Acceleration

MetaSelector: Meta-Learning for Recommendation with User-Level Adaptive Model Selection

Green Runner: A tool for efficient deep learning component selection

MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuning

Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs

Neuro-Symbolic Embedding for Short and Effective Feature Selection via Autoregressive Generation