Abstract:Fine-tuning techniques based on Large Pretrained Language Models (LPLMs) have been proven to significantly enhance model performance on a variety of downstream tasks and effectively control the output behaviors of LPLMs. Recent studies have proposed numerous methods for fine-tuning a small number of parameters based on open-source LPLMs, reducing the demand for computational and storage resources. Among these, reparameterization fine-tuning methods represented by LoRA (Low-Rank Adaptation) have gained popularity. We find that although these methods perform well in many aspects, there is still considerable room for improvement in terms of complex task adaptability, performance, stability, and algorithm complexity. In response to this, inspired by the idea that the functions of the brain are shaped by its geometric structure, this paper integrates this idea into LoRA technology and proposes a new matrix transformation-based reparameterization method for efficient fine-tuning, named Matrix-Transformation based Low-Rank Adaptation (MTLoRA). MTLoRA aims to dynamically alter its spatial geometric structure by applying a transformation-matrix T to perform linear transformations, such as rotation, scaling, and translation, on the task-specific parameter matrix, generating new matrix feature patterns (eigenvectors) to mimic the fundamental influence of complex geometric structure feature patterns in the brain on functions, thereby enhancing the model's performance in downstream tasks. In Natural Language Understanding (NLU) tasks, it is evaluated using the GLUE benchmark test, and the results reveal that MTLoRA achieves an overall performance increase of about 1.0% across eight tasks; in Natural Language Generation (NLG) tasks, MTLoRA improves performance by an average of 0.95% and 0.56% in the DART and WebNLG tasks, respectively.

Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning

Is Parameter Collision Hindering Continual Learning in LLMs?

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

Full Parameter Fine-tuning for Large Language Models with Limited Resources

Rehearsal-free Continual Language Learning via Efficient Parameter Isolation

Scaling Laws for Forgetting When Fine-Tuning Large Language Models

OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning

LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning

Revisiting Catastrophic Forgetting in Large Language Model Tuning

Parameter-efficient Tuning for Large Language Model Without Calculating Its Gradients

CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation

SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture

Unlocking Continual Learning Abilities in Language Models

MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models

LoRA-Mini : Adaptation Matrices Decomposition and Selective Training

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models

Learning Global Controller in Latent Space for Parameter-Efficient Fine-Tuning

Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning