Abstract:Adapters have been widely explored to alleviate computational and storage costs when fine-tuning pretrained foundation models. However, the adapter itself can exhibit redundancy, leading to unnecessary storage overhead and inferior performance. In this paper, we propose Prune and Share (Pear), a novel adapter-pruning framework for efficient fine-tuning of pretrained visual foundation models. Specifically, we prune certain adapters and share the more important unpruned ones with positions where adapters are pruned, allowing continual adaptation at these positions after pruning. Additionally, a knowledge checkpoint strategy is introduced, which preserves the information of the pruned adapters and further boosts performance. Experimental results on visual adaptation benchmark validate the effectiveness and efficiency of the proposed Pear comparing to other competitive methods. Code is in <a class="link-external link-https" href="https://github.com/yibozhong/pear" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: when fine - tuning pre - trained visual foundation models, how to reduce redundant parameters through pruning and sharing adapters, thereby reducing computational and storage costs while maintaining or improving model performance. Specifically, although existing adapters can significantly reduce the number of parameters and GPU memory usage, they may still be redundant themselves, resulting in unnecessary storage overhead and performance degradation. To solve this problem, the authors propose the Prune and Share (Pear) method, which aims to achieve efficient fine - tuning by pruning unimportant adapters and sharing important ones. ### Main problem summary: 1. **Redundant adapters**: Existing adapters are themselves redundant, resulting in unnecessary storage overhead and performance degradation. 2. **Limitations of traditional pruning methods**: Traditional structured pruning methods directly remove adapters, causing the pruned positions to be unable to continue adapting, affecting the overall performance. 3. **Information loss**: Directly pruning adapters will discard the information they contain. Even if these adapters are relatively unimportant, retaining this information is still helpful for more comprehensive adaptation. ### Solutions: - **Prune and Share (Pear)**: By pruning unimportant adapters and sharing the remaining important adapters to the pruned positions, all positions can continue to adapt without introducing additional parameters. - **Knowledge Checkpoint**: Retain the information of the pruned adapters to further improve the adaptation effect. Through these methods, Pear can reduce the number of parameters while maintaining or even improving the model performance, especially in resource - constrained situations (such as limited GPU memory or lightweight devices).

Pear: Pruning and Sharing Adapters in Visual Parameter-Efficient Fine-Tuning

Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision

Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy

Parameter-Efficient Fine-Tuning With Adapters

SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters

Sheared Backpropagation for Fine-Tuning Foundation Models

MoSA: Mixture of Sparse Adapters for Visual Efficient Tuning

Split & Merge: Unlocking the Potential of Visual Adapters via Sparse Training

VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

AiRs: Adapter in Remote Sensing for Parameter-Efficient Transfer Learning

Not All Data Matters: An End-to-End Adaptive Dataset Pruning Framework for Enhancing Model Performance and Efficiency

Time-, Memory- and Parameter-Efficient Visual Adaptation

Adaptable Adapters

Non-Parametric Adaptive Network Pruning

AdaPruner: Adaptive Channel Pruning and Effective Weights Inheritance

Adapter Pruning using Tropical Characterization

Tuning Vision-Language Models with Multiple Prototypes Clustering

Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictions

On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Network Pruning Using Adaptive Exemplar Filters