Abstract:Low-rank adaptations (LoRA) are widely used to fine-tune large models across various domains for specific downstream tasks. While task-specific LoRAs are often available, concerns about data privacy and intellectual property can restrict access to training data, limiting the acquisition of a multi-task model through gradient-based training. In response, LoRA merging presents an effective solution by combining multiple LoRAs into a unified adapter while maintaining data privacy. Prior works on LoRA merging primarily frame it as an optimization problem, yet these approaches face several limitations, including the rough assumption about input features utilized in optimization, massive sample requirements, and the unbalanced optimization objective. These limitations can significantly degrade performance. To address these, we propose a novel optimization-based method, named IterIS: 1) We formulate LoRA merging as an advanced optimization problem to mitigate the rough assumption. Additionally, we employ an iterative inference-solving framework in our algorithm. It can progressively refine the optimization objective for improved performance. 2) We introduce an efficient regularization term to reduce the need for massive sample requirements (requiring only 1-5% of the unlabeled samples compared to prior methods). 3) We utilize adaptive weights in the optimization objective to mitigate potential unbalances in LoRA merging process. Our method demonstrates significant improvements over multiple baselines and state-of-the-art methods in composing tasks for text-to-image diffusion, vision-language models, and large language models. Furthermore, our layer-wise algorithm can achieve convergence with minimal steps, ensuring efficiency in both memory and computation.

LoRA Fusion: Enhancing Image Generation

CLoRA: A Contrastive Approach to Compose Multiple LoRA Models

Multi-LoRA Composition for Image Generation

LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation

IterIS: Iterative Inference-Solving Alignment for LoRA Merging

LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models

HyperLoRA: Efficient Cross-task Generalization Via Constrained Low-Rank Adapters Generation

Mixture of LoRA Experts

LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks

LoRA Diffusion: Zero-Shot LoRA Synthesis for Diffusion Model Personalization

TriLoRA: Integrating SVD for Advanced Style Personalization in Text-to-Image Generation

DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion

In-Context LoRA for Diffusion Transformers

V-LoRA: an Efficient and Flexible System Boosts Vision Applications with LoRA LMM

DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models

SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis

ResLoRA: Identity Residual Mapping in Low-Rank Adaption

FLoCoRA: Federated learning compression with low-rank adaptation

Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generation

mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs

LCM-LoRA: A Universal Stable-Diffusion Acceleration Module