Abstract:Fine-tuning large pre-trained language models on downstream tasks has become the de-facto learning paradigm in NLP. However, conventional approaches fine-tune all the parameters of the pre-trained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. In this paper, we break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them. Specifically, we re-frame them as modifications to specific hidden states in pre-trained models, and define a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Through comprehensive empirical studies across machine translation, text summarization, language understanding, and text classification benchmarks, we utilize the unified view to identify important design choices in previous methods. Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks.

Optimal Parameter-Transfer Learning by Semiparametric Model Averaging

Model Averaging Estimation for Nonparametric Varying-Coefficient Models with Multiplicative Heteroscedasticity

Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer.

AdaBoost Semiparametric Model Averaging Prediction for Multiple Categories

Model averaging estimation for generalized partially linear varying-coefficient models

Optimal Model Averaging for Semiparametric Partially Linear Models with Censored Data

Penalized Time-Varying Model Averaging

Parsimonious Model Averaging With a Diverging Number of Parameters

Model Averaging for Semiparametric Varying Coefficient Quantile Regression Models

One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging For Cross-Lingual Transfer

Towards a Unified View of Parameter-Efficient Transfer Learning

Partial Linear Model Averaging Prediction for Longitudinal Data

Jackknife Model Averaging for Additive Expectile Prediction

Representation Transfer Learning for Semiparametric Regression

Model averaging prediction by K -fold cross-validation

Semiparametric model averaging for high dimensional conditional quantile prediction

Optimal Model Averaging for Semiparametric Partially Linear Models with Measurement Errors

Model-averaging-based semiparametric modeling for conditional quantile prediction

Parameter Transfer Extreme Learning Machine based on Projective Model

Model Averaging by Cross-validation for Partially Linear Functional Additive Models

Prediction Using Many Samples with Models Possibly Containing Partially Shared Parameters