Abstract:Fine-tuning techniques based on Large Pretrained Language Models (LPLMs) have been proven to significantly enhance model performance on a variety of downstream tasks and effectively control the output behaviors of LPLMs. Recent studies have proposed numerous methods for fine-tuning a small number of parameters based on open-source LPLMs, reducing the demand for computational and storage resources. Among these, reparameterization fine-tuning methods represented by LoRA (Low-Rank Adaptation) have gained popularity. We find that although these methods perform well in many aspects, there is still considerable room for improvement in terms of complex task adaptability, performance, stability, and algorithm complexity. In response to this, inspired by the idea that the functions of the brain are shaped by its geometric structure, this paper integrates this idea into LoRA technology and proposes a new matrix transformation-based reparameterization method for efficient fine-tuning, named Matrix-Transformation based Low-Rank Adaptation (MTLoRA). MTLoRA aims to dynamically alter its spatial geometric structure by applying a transformation-matrix T to perform linear transformations, such as rotation, scaling, and translation, on the task-specific parameter matrix, generating new matrix feature patterns (eigenvectors) to mimic the fundamental influence of complex geometric structure feature patterns in the brain on functions, thereby enhancing the model's performance in downstream tasks. In Natural Language Understanding (NLU) tasks, it is evaluated using the GLUE benchmark test, and the results reveal that MTLoRA achieves an overall performance increase of about 1.0% across eight tasks; in Natural Language Generation (NLG) tasks, MTLoRA improves performance by an average of 0.95% and 0.56% in the DART and WebNLG tasks, respectively.

HyperLoRA: Efficient Cross-task Generalization Via Constrained Low-Rank Adapters Generation

LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

MultiLoRA: Democratizing LoRA for Better Multi-Task Learning

SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules

Task-Agnostic Low-Rank Adapters for Unseen English Dialects

Instance-Level Dynamic LoRAs Composition for Cross-Task Generalization

Batched Low-Rank Adaptation of Foundation Models

Chain-of-LoRA: Enhancing the Instruction Fine-Tuning Performance of Low-Rank Adaptation on Diverse Instruction Set

Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning

Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning

GeoLoRA: Geometric integration for parameter efficient fine-tuning

Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learning

ResLoRA: Identity Residual Mapping in Low-Rank Adaption

SwitchLoRA: Switched Low-Rank Adaptation Can Learn Full-Rank Information

GraphLoRA: Structure-Aware Contrastive Low-Rank Adaptation for Cross-Graph Transfer Learning

LoRA: Low-Rank Adaptation of Large Language Models

Flexora: Flexible Low Rank Adaptation for Large Language Models

BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models

A Survey on LoRA of Large Language Models