Abstract:Domain generalization (DG) aims to avoid the performance degradation of the model when the distribution shift between the limited training data and unseen test data occurs. Recently, foundation models with enormous parameters have been pre-trained with huge datasets, demonstrating strong generalization ability and showing promising direction for solving the DG problem. However, fully Fine-Tuning (FT) the foundation models results in unsatisfactory out-of-distribution accuracy due to the destroyed pre-trained generalized features. Recently, Parameter-Efficient Fine-Tuning (PEFT) alleviates the above problem by fine-tuning a small portion of the model parameters while keeping the rest frozen, which achieves better generalization performance compared to FT. Nevertheless, PEFT still suffers from the issue of overfitting to the training domains. To address the above issue, we propose Parameter-Efficient Group with Orthogonal regularization (PEGO) for vision transformers, which effectively preserves the generalization ability of the pre-trained network and learns more diverse knowledge compared with conventional PEFT. Specifically, we inject a group of trainable Low-Rank Adaptation (LoRA) modules into the pre-trained model and propose an orthogonal regularization loss to enhance the generalization ability of the model. Our framework achieves SOTA performance on five DG benchmarks, while only requiring training a small number of parameters without adding additional testing cost.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve two key problems in **Domain Generalization (DG)**: 1. **The over - fitting problem during fine - tuning of pre - trained models**: - When using large - scale pre - trained models (such as Vision Transformer, ViT) for fine - tuning, directly performing full - scale fine - tuning (Full Fine - Tuning, FT) will lead to a decline in the performance of the model on unseen test data. This is because when training on limited source - domain data, a large number of parameters are prone to cause over - fitting. - Although Parameter - Efficient Fine - Tuning (PEFT) alleviates the over - fitting problem by only fine - tuning a small number of parameters, there is still a risk of over - fitting to the source domain, and it may partially distort the generalization characteristics of the pre - trained model. 2. **How to make full use of the generalization ability of pre - trained models**: - After pre - training on large - scale data sets, large - scale pre - trained models have strong generalization abilities. However, in DG tasks, how to preserve and make full use of these generalization abilities during the fine - tuning process is a challenge. - Existing DG methods mainly focus on how to extract invariant features from limited source domains or generate more training data through data augmentation, while ignoring how to preserve and utilize the generalization ability of the pre - trained model itself. To solve the above problems, the authors propose the **Parameter - Efficient Group with Orthogonal Regularization (PEGO)** framework. Specifically: - **Learn to Preserve**: By introducing an orthogonal regularization loss, the weights of the injected LoRA module are constrained to be orthogonal to the pre - trained weights, thereby minimizing the distortion of the pre - trained generalization features and preserving the generalization ability of the pre - trained model. - **Learn to Diversify**: By injecting multiple LoRA modules in each layer and imposing orthogonal constraints between these modules, the model is encouraged to learn more diverse knowledge in order to better handle various unseen domains. Through these two mechanisms, PEGO can not only effectively alleviate the over - fitting problem, but also significantly improve the generalization performance of the model in unseen domains. Experimental results show that PEGO has achieved state - of - the - art performance on multiple DG benchmark data sets.

Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization

Domain Generalization Using Large Pretrained Models with Mixture-of-Adapters

SoRA: Singular Value Decomposed Low-Rank Adaptation for Domain Generalizable Representation Learning

Domain Generalization Guided by Large-Scale Pre-Trained Priors

Parameter Exchange for Robust Dynamic Domain Generalization

Towards Unified and Effective Domain Generalization

PACE: marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization

Trainable Projected Gradient Method for Robust Fine-tuning

Adaptive Principal Components Allocation with the $\ell_{2,g}$-regularized Gaussian Graphical Model for Efficient Fine-Tuning Large Models

More is Better: A Novel Multi-view Framework for Domain Generalization

Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies

Gradient Projection For Continual Parameter-Efficient Tuning

Federated Domain Generalization with Generalization Adjustment

MVDG: A Unified Multi-view Framework for Domain Generalization

Diverse Target and Contribution Scheduling for Domain Generalization

DARA: Domain- and Relation-aware Adapters Make Parameter-efficient Tuning for Visual Grounding

See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition

Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning

PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization

Dual Low-Rank Adaptation for Continual Learning with Pre-Trained Models

ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts