LoTR: Low Tensor Rank Weight Adaptation

Daniel Bershatsky,Daria Cherniuk,Talgat Daulbaev,Aleksandr Mikhalev,Ivan Oseledets

2024-02-05

Abstract:In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture. Widely used LoRA-like methods of fine-tuning LLMs are based on matrix factorization of gradient update. We introduce LoTR, a novel approach for parameter-efficient fine-tuning of LLMs which represents a gradient update to parameters in a form of tensor decomposition. Low-rank adapter for each layer is constructed as a product of three matrices, and tensor structure arises from sharing left and right multipliers of this product among layers. Simultaneous compression of a sequence of layers with low-rank tensor representation allows LoTR to archive even better parameter efficiency then LoRA especially for deep models. Moreover, the core tensor does not depend on original weight dimension and can be made arbitrary small, which allows for extremely cheap and fast downstream fine-tuning.

Computation and Language,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The paper introduces LoTR (Low Tensor Rank Weight Adaptation), which is a novel approach for parameter-efficient fine-tuning of large language models (LLMs) based on the Transformer architecture. The primary problem addressed by this work is the high computational cost associated with fine-tuning large pre-trained language models when the number of parameters grows significantly.

Key points of the approach:
- **Objective**: To reduce the number of trainable parameters required during fine-tuning while maintaining or improving model performance.
- **Technique**: LoTR uses low-rank tensor decomposition to represent gradient updates in a more compact form compared to existing methods like LoRA (Low-Rank Adaptation).
- **Methodology**: LoTR constructs a low-rank adapter for each layer as a product of three matrices, where the left and right multipliers are shared among layers, forming a tensor structure. This allows for simultaneous compression of a sequence of layers using low-rank tensor representation.
- **Advantages**: Compared to LoRA, LoTR achieves better parameter efficiency, especially for deep models. The core tensor in LoTR does not depend on the original weight dimensions and can be made arbitrarily small, enabling extremely cheap and fast downstream fine-tuning.
- **Empirical Results**: The paper demonstrates that LoTR performs competitively with LoRA and achieves similar performance on downstream tasks. Additionally, ablation studies provide insights into hyperparam