Abstract:Adapting pre-trained foundation models for various downstream tasks has been prevalent in artificial intelligence. Due to the vast number of tasks and high costs, adjusting all parameters becomes unfeasible. To mitigate this, several fine-tuning techniques have been developed to update the pre-trained model weights in a more resource-efficient manner, such as through low-rank adjustments. Yet, almost all of these methods focus on linear weights, neglecting the intricacies of parameter spaces in higher dimensions like 4D. Alternatively, some methods can be adapted for high-dimensional parameter space by compressing changes in the original space into two dimensions and then employing low-rank matrix decomposition. However, these approaches destructs the structural integrity of the involved high-dimensional spaces. To tackle the diversity of dimensional spaces across different foundation models and provide a more precise representation of the changes within these spaces, this paper introduces a generalized parameter-efficient fine-tuning framework, FLoRA, designed for various dimensional parameter space. Specifically, utilizing Tucker decomposition, FLoRA asserts that changes in each dimensional parameter space are based on a low-rank core space which maintains the consistent topological structure with the original space. It then models the changes through this core space alongside corresponding weights to reconstruct alterations in the original space. FLoRA effectively preserves the structural integrity of the change of original N-dimensional parameter space, meanwhile decomposes it via low-rank tensor decomposition. Extensive experiments on computer vision, natural language processing and multi-modal tasks validate FLoRA's effectiveness. Codes are available at

What problem does this paper attempt to address?

The paper aims to address the issues encountered when fine-tuning large-scale pre-trained models on downstream tasks, especially when the model parameters are enormous, making it extremely expensive and impractical to fine-tune all parameters completely. To tackle this challenge, the researchers propose a new method called FLoRA (Fundamental LOw-Rank Adaptation). Specifically, FLoRA aims to achieve efficient parameter fine-tuning in different dimensions of the parameter space while preserving the topological structure of these spaces. Existing methods mainly focus on 2-dimensional linear layers, neglecting higher-dimensional spaces (such as 4-dimensional convolutional layers). These methods typically handle changes in high-dimensional parameter spaces through low-rank matrix decomposition, but this approach disrupts the locality and structural integrity of the original space. FLoRA utilizes Tucker decomposition to retain the structural integrity in high-dimensional parameter spaces and can effectively represent these changes. The main contributions of the paper include: 1. Proposing a new Parameter-Efficient Fine-Tuning (PEFT) method, FLoRA, which is suitable for different dimensions of parameter spaces and aims to seek low-rank representations while preserving their topological structure. 2. Extensive experiments on computer vision, natural language processing, and multimodal tasks demonstrate that FLoRA significantly outperforms other baseline methods, validating its effectiveness.

FLoRA: Low-Rank Core Space for N-dimension

Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape

CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models

Parameter-Efficient Fine-Tuning with Discrete Fourier Transform

Batched Low-Rank Adaptation of Foundation Models

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement

NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models

LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

ResLoRA: Identity Residual Mapping in Low-Rank Adaption

Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models

LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

LoTR: Low Tensor Rank Weight Adaptation

Improving LoRA in Privacy-preserving Federated Learning

IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning

PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation

AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models

FLoCoRA: Federated learning compression with low-rank adaptation

Flexora: Flexible Low Rank Adaptation for Large Language Models

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation