SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning

Haiwen Diao,Bo Wan,Xu Jia,Yunzhi Zhuge,Ying Zhang,Huchuan Lu,Long Chen

2024-07-10

Abstract:Parameter-efficient transfer learning (PETL) has emerged as a flourishing research field for adapting large pre-trained models to downstream tasks, greatly reducing trainable parameters while grappling with memory challenges during fine-tuning. To address it, memory-efficient series (METL) avoid backpropagating gradients through the large backbone. However, they compromise by exclusively relying on frozen intermediate outputs and limiting the exhaustive exploration of prior knowledge from pre-trained models. Moreover, the dependency and redundancy between cross-layer features are frequently overlooked, thereby submerging more discriminative representations and causing an inherent performance gap (vs. conventional PETL methods). Hence, we propose an innovative METL strategy called SHERL for resource-limited scenarios to decouple the entire adaptation into two successive and complementary processes. In the early route, intermediate outputs are consolidated via an anti-redundancy operation, enhancing their compatibility for subsequent interactions; thereby in the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead and regulate these fairly flexible features into more adaptive and powerful representations for new domains. Extensive ablations on vision-and-language and language-only tasks show that SHERL combines the strengths of both parameter and memory-efficient techniques, performing on-par or better across diverse architectures with lower memory during fine-tuning. Our code is publicly available at: <a class="link-external link-https" href="https://github.com/Paranioar/SHERL" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Multimedia

What problem does this paper attempt to address?

The paper aims to address key issues in Parameter-Efficient Transfer Learning (PETL) and Memory-Efficient Transfer Learning (METL), particularly how to optimize the fine-tuning process of large pre-trained models under resource-constrained conditions. Specifically:

1. **Reducing Memory Overhead**: Existing PETL methods reduce the number of training parameters but still require the transmission of a large number of gradients during fine-tuning, leading to high memory overhead. While METL methods attempt to reduce memory consumption through bypass networks, they overlook the redundancy and dependencies between cross-layer features.

2. **Improving Performance**: Existing METL methods only utilize frozen intermediate outputs, limiting the full utilization of prior knowledge from pre-trained models. Additionally, they fail to effectively handle the redundancy of cross-layer features, resulting in insufficient discriminative power of the final representation.

To address the above issues, the paper proposes a new method called SHERL, which combines early aggregation and late adjustment to improve the model's adaptability and accuracy on new tasks while maintaining low memory consumption. SHERL performs well on both vision-language tasks and pure language tasks and can seamlessly integrate with other PETL methods, making it suitable for various network architectures.