Abstract:Sequential recommendation systems predict the next interaction item based on users' past interactions, aligning recommendations with individual preferences. Leveraging the strengths of Large Language Models (LLMs) in knowledge comprehension and reasoning, recent approaches are eager to apply LLMs to sequential recommendation. A common paradigm is converting user behavior sequences into instruction data, and fine-tuning the LLM with parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaption (LoRA). However, the uniform application of LoRA across diverse user behaviors is insufficient to capture individual variability, resulting in negative transfer between disparate sequences. To address these challenges, we propose Instance-wise LoRA (iLoRA). We innovatively treat the sequential recommendation task as a form of multi-task learning, integrating LoRA with the Mixture of Experts (MoE) framework. This approach encourages different experts to capture various aspects of user behavior. Additionally, we introduce a sequence representation guided gate function that generates customized expert participation weights for each user sequence, which allows dynamic parameter adjustment for instance-wise recommendations. In sequential recommendation, iLoRA achieves an average relative improvement of 11.4\% over basic LoRA in the hit ratio metric, with less than a 1\% relative increase in trainable parameters. Extensive experiments on three benchmark datasets demonstrate the effectiveness of iLoRA, highlighting its superior performance compared to existing methods in mitigating negative transfer and improving recommendation accuracy. Our data and code are available at <a class="link-external link-https" href="https://github.com/AkaliKong/iLoRA" rel="external noopener nofollow">this https URL</a>.

Instance-Level Dynamic LoRAs Composition for Cross-Task Generalization

LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

HyperLoRA: Efficient Cross-task Generalization Via Constrained Low-Rank Adapters Generation

LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks

LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild

LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks

DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models

Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models

MultiLoRA: Democratizing LoRA for Better Multi-Task Learning

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

IterIS: Iterative Inference-Solving Alignment for LoRA Merging

A Framework to Implement 1+N Multi-task Fine-tuning Pattern in LLMs Using the CGC-LORA Algorithm

SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules

Mixture of LoRA Experts

Customizing Language Models with Instance-wise LoRA for Sequential Recommendation

CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models

Chain-of-LoRA: Enhancing the Instruction Fine-Tuning Performance of Low-Rank Adaptation on Diverse Instruction Set

ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers

AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality

Multi-LoRA Composition for Image Generation

LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models