Abstract: It is common within the deep learning community to first pre-train a deep neural network from a large-scale dataset and then fine-tune the pre-trained model to a specific downstream task. Recently, both supervised and unsupervised pre-training approaches to learning representations have achieved remarkable advances, which exploit the discriminative knowledge of labels and the intrinsic structure of data, respectively. It follows natural intuition that both discriminative knowledge and intrinsic structure of the downstream task can be useful for fine-tuning, however, existing fine-tuning methods mainly leverage the former and discard the latter. A question arises: How to fully explore the intrinsic structure of data for boosting fine-tuning? In this paper, we propose Bi-tuning, a general learning framework to fine-tuning both supervised and unsupervised pre-trained representations to downstream tasks. Bi-tuning generalizes the vanilla fine-tuning by integrating two heads upon the backbone of pre-trained representations: a classifier head with an improved contrastive cross-entropy loss to better leverage the label information in an instance-contrast way, and a projector head with a newly-designed categorical contrastive learning loss to fully exploit the intrinsic structure of data in a category-consistent way. Comprehensive experiments confirm that Bi-tuning achieves state-of-the-art results for fine-tuning tasks of both supervised and unsupervised pre-trained models by large margins (e.g. 10.7\% absolute rise in accuracy on CUB in low-data regime).

Bi-tuning of Pre-trained Representations

Bi-tuning: Efficient Transfer from Pre-trained Models

Co-Tuning for Transfer Learning.

Improved Fine-tuning by Leveraging Pre-training Data: Theory and Practice

Muppet: Massive multi-task representations with pre-finetuning

Improved Visual Fine-tuning with Natural Language Supervision

Improved Fine-Tuning by Better Leveraging Pre-Training Data

BiSSL: Bilevel Optimization for Self-Supervised Pre-Training and Fine-Tuning

KNN-BERT: Fine-Tuning Pre-Trained Models with KNN Classifier

DR-Tune: Improving Fine-tuning of Pretrained Visual Models by Distribution Regularization with Semantic Calibration

A Closer Look at How Fine-tuning Changes BERT

Parameter-Efficient Tuning Makes a Good Classification Head.

Idling Neurons, Appropriately Lenient Workload During Fine-tuning Leads to Better Generalization

Investigating Learning Dynamics of BERT Fine-Tuning

Effective Domain Knowledge Transfer with Soft Fine-tuning

Multifaceted Analysis of Fine-Tuning in Deep Model for Visual Recognition

Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

Two-Stage Fine-Tuning: A Novel Strategy for Learning Class-Imbalanced Data

Why pre-training is beneficial for downstream classification tasks?

Layer-wise Learning Rate Optimization for Task-Dependent Fine-Tuning of Pre-trained Models: An Evolutionary Approach

Visualizing and Understanding the Effectiveness of BERT.