Abstract:Fine-tuning techniques based on Large Pretrained Language Models (LPLMs) have been proven to significantly enhance model performance on a variety of downstream tasks and effectively control the output behaviors of LPLMs. Recent studies have proposed numerous methods for fine-tuning a small number of parameters based on open-source LPLMs, reducing the demand for computational and storage resources. Among these, reparameterization fine-tuning methods represented by LoRA (Low-Rank Adaptation) have gained popularity. We find that although these methods perform well in many aspects, there is still considerable room for improvement in terms of complex task adaptability, performance, stability, and algorithm complexity. In response to this, inspired by the idea that the functions of the brain are shaped by its geometric structure, this paper integrates this idea into LoRA technology and proposes a new matrix transformation-based reparameterization method for efficient fine-tuning, named Matrix-Transformation based Low-Rank Adaptation (MTLoRA). MTLoRA aims to dynamically alter its spatial geometric structure by applying a transformation-matrix T to perform linear transformations, such as rotation, scaling, and translation, on the task-specific parameter matrix, generating new matrix feature patterns (eigenvectors) to mimic the fundamental influence of complex geometric structure feature patterns in the brain on functions, thereby enhancing the model's performance in downstream tasks. In Natural Language Understanding (NLU) tasks, it is evaluated using the GLUE benchmark test, and the results reveal that MTLoRA achieves an overall performance increase of about 1.0% across eight tasks; in Natural Language Generation (NLG) tasks, MTLoRA improves performance by an average of 0.95% and 0.56% in the DART and WebNLG tasks, respectively.

Vertical LoRA: Dense Expectation-Maximization Interpretation of Transformers

Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA

LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks

AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers

FullLoRA-AT: Efficiently Boosting the Robustness of Pretrained Vision Transformers

Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning

Computational Limits of Low-Rank Adaptation (LoRA) for Transformer-Based Models

On the Long Range Abilities of Transformers

The Expressive Power of Low-Rank Adaptation

Interpret Vision Transformers as ConvNets with Dynamic Convolutions

LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking

Higher Layers Need More LoRA Experts

LoTR: Low Tensor Rank Weight Adaptation

VisionLLaMA: A Unified LLaMA Backbone for Vision Tasks

LoRA: Low-Rank Adaptation of Large Language Models

Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape

Linear Transformers are Versatile In-Context Learners

The Impact of LoRA on the Emergence of Clusters in Transformers

Analyzing Transformers in Embedding Space

Deep Transformers with Latent Depth