Abstract:Fine-tuning techniques based on Large Pretrained Language Models (LPLMs) have been proven to significantly enhance model performance on a variety of downstream tasks and effectively control the output behaviors of LPLMs. Recent studies have proposed numerous methods for fine-tuning a small number of parameters based on open-source LPLMs, reducing the demand for computational and storage resources. Among these, reparameterization fine-tuning methods represented by LoRA (Low-Rank Adaptation) have gained popularity. We find that although these methods perform well in many aspects, there is still considerable room for improvement in terms of complex task adaptability, performance, stability, and algorithm complexity. In response to this, inspired by the idea that the functions of the brain are shaped by its geometric structure, this paper integrates this idea into LoRA technology and proposes a new matrix transformation-based reparameterization method for efficient fine-tuning, named Matrix-Transformation based Low-Rank Adaptation (MTLoRA). MTLoRA aims to dynamically alter its spatial geometric structure by applying a transformation-matrix T to perform linear transformations, such as rotation, scaling, and translation, on the task-specific parameter matrix, generating new matrix feature patterns (eigenvectors) to mimic the fundamental influence of complex geometric structure feature patterns in the brain on functions, thereby enhancing the model's performance in downstream tasks. In Natural Language Understanding (NLU) tasks, it is evaluated using the GLUE benchmark test, and the results reveal that MTLoRA achieves an overall performance increase of about 1.0% across eight tasks; in Natural Language Generation (NLG) tasks, MTLoRA improves performance by an average of 0.95% and 0.56% in the DART and WebNLG tasks, respectively.

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning

MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models

ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models

MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning

Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning

MoR: Mixture of Ranks for Low-Rank Adaptation Tuning

Higher Layers Need More LoRA Experts

mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs

Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models

MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models

DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models

IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning

FanLoRA: Fantastic LoRAs and Where to Find Them in Large Language Model Fine-tuning

MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

GraphLoRA: Empowering LLMs Fine-Tuning via Graph Collaboration of MoE

AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality

MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning