Abstract:Large Language Models (LLMs), built on Transformer architectures, exhibit remarkable generalization across a wide range of tasks. However, fine-tuning these models for specific tasks remains resource-intensive due to their extensive parameterization. In this paper, we investigate two remarkable phenomena observed during the fine-tuning of LLMs, particularly focusing on the attention mechanism: (1) Different Impact, optimizing the $\mathbf{W}_v$ matrix significantly improves performance over optimizing the $\mathbf{W}_k$ matrix. Fine-tuning only the $\mathbf{W}_q$ and $\mathbf{W}_v$ matrices is computationally efficient, delivering results that are comparable to, or even better than, fine-tuning all three matrices $\mathbf{W}_q$, $\mathbf{W}_k$, and $\mathbf{W}_v$. (2) Efficient Convergence, employing distinct learning rates for these matrices is crucial for optimal performance, with a higher learning rate for the $\mathbf{W}_v$ matrix expediting convergence. However, theoretical analyses of these phenomena are still relatively limited. We present a theoretical analysis of these phenomena from two perspectives: (i) Generalization, where we demonstrate that fine-tuning only $\mathbf{W}_q$ and $\mathbf{W}_v$ improves generalization bounds, enhances memory efficiency, and (ii) Optimization, where we emphasize that the feature learning of the attention mechanism is efficient, particularly when using distinct learning rates for the matrices, which leads to more effective fine-tuning. Building on these insights, we propose a new strategy that improves fine-tuning efficiency in terms of both storage and time. Experimental results on benchmark datasets validate the effectiveness of this approach, supporting our theoretical findings. Our analysis lays the theoretical groundwork for configuring and improving lightweight algorithms in LLMs fine-tuning.

Reparameterized Attention for Convolutional Neural Networks

An Attention Module for Convolutional Neural Networks

Generalized Parameter Estimation Method for Model-Based Real Time Optimization

Less is More Important: an Attention Module Guided by Probability Density Function for Convolutional Neural Networks.

A Generalized Attention Mechanism to Enhance the Accuracy Performance of Neural Networks

Convolutional Neural Network optimization via Channel Reassessment Attention module

HAM: Hybrid Attention Module in Deep Convolutional Neural Networks for Image Classification

Bayesian Attention Belief Networks

A Multi-scale Contextual Attention Mechanism for Convolutional Neural Networks

Attention-enhanced neural network models for turbulence simulation

Multiscale Hybrid Convolutional Deep Neural Networks with Channel Attention

Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization

Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

Parameter-Free Channel Attention for Image Classification and Super-Resolution

End-to-End Conformal Calibration for Optimization Under Uncertainty

Learning Wake-Sleep Recurrent Attention Models

ATCON: Attention Consistency for Vision Models

A feature-wise attention module based on the difference with surrounding features for convolutional neural networks

Attending Via Both Fine-tuning and Compressing.

Exploring the Power of Pure Attention Mechanisms in Blind Room Parameter Estimation

Improving model calibration with accuracy versus uncertainty optimization