Abstract:Multimodal foundation models are transformative in sequential recommender systems, leveraging powerful representation learning capabilities. While Parameter-efficient Fine-tuning (PEFT) is commonly used to adapt foundation models for recommendation tasks, most research prioritizes parameter efficiency, often overlooking critical factors like GPU memory efficiency and training speed. Addressing this gap, our paper introduces IISAN (Intra- and Inter-modal Side Adapted Network for Multimodal Representation), a simple plug-and-play architecture using a Decoupled PEFT structure and exploiting both intra- and inter-modal adaptation. IISAN matches the performance of full fine-tuning (FFT) and state-of-the-art PEFT. More importantly, it significantly reduces GPU memory usage - from 47GB to just 3GB for multimodal sequential recommendation tasks. Additionally, it accelerates training time per epoch from 443s to 22s compared to FFT. This is also a notable improvement over the Adapter and LoRA, which require 37-39 GB GPU memory and 350-380 seconds per epoch for training. Furthermore, we propose a new composite efficiency metric, TPME (Training-time, Parameter, and GPU Memory Efficiency) to alleviate the prevalent misconception that "parameter efficiency represents overall efficiency". TPME provides more comprehensive insights into practical efficiency comparisons between different methods. Besides, we give an accessible efficiency analysis of all PEFT and FFT approaches, which demonstrate the superiority of IISAN. We release our codes and other materials at <a class="link-external link-https" href="https://github.com/GAIR-Lab/IISAN" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is how to efficiently adapt large - scale pre - trained models (i.e., base models) in multi - modal sequence recommendation tasks, so as to significantly reduce GPU memory usage and accelerate training time while maintaining high performance. Specifically: 1. **Misunderstanding of Parameter - efficiency and Practical - efficiency**: Many studies mainly focus on parameter - efficient fine - tuning (PEFT), but ignore practical efficiency, such as GPU memory usage and training speed. The paper points out that parameter - efficiency is not directly equivalent to overall efficiency. 2. **Limitations of Existing PEFT Methods**: Although existing embedded PEFT methods (such as Adapter and LoRA) reduce the number of trainable parameters, the computational graph is still heavy during the back - propagation process, resulting in no significant reduction in GPU memory consumption and training time. 3. **Challenges of Multi - modal Recommendation Tasks**: Multi - modal recommendation tasks combine text and image information, which greatly increases the model scale and data input, and further exacerbates the problems of GPU memory and training time. To solve these problems, the paper proposes the following innovations: - **IISAN Architecture**: A new Intra - and Inter - modal Side Adapted Network (IISAN) architecture is introduced. It adopts a decoupled PEFT (DPEFT) structure. By separating the new trainable side - adapted network (SAN) and the frozen base model, the computational graph in back - propagation is optimized, thereby significantly reducing GPU memory usage and accelerating training time. - **Caching Strategy**: Taking advantage of DPEFT, a caching strategy is proposed to store and reuse the item latent states extracted from the pre - trained multi - modal base model, avoiding repeated forward - propagation and further improving efficiency. - **New Efficiency Metric TPME**: A comprehensive efficiency metric TPME (Training - time, Parameter, and GPU Memory Efficiency) is proposed, which incorporates three key factors, namely training time, trainable parameters, and GPU memory usage, into a unified evaluation index, so as to more comprehensively evaluate the practical efficiency of different methods. Through these innovations, IISAN is not only comparable in performance to full fine - tuning (FFT) and the state - of - the - art PEFT methods, but also has achieved significant improvements in GPU memory usage and training time.

IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT

Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation

MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter

A Flexible Embedding-Aware Near Memory Processing Architecture for Recommendation System

POSTER: Pattern-Aware Sparse Communication for Scalable Recommendation Model Training.

Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision

Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation

AiRs: Adapter in Remote Sensing for Parameter-Efficient Transfer Learning

FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

M$^2$IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension

PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation

See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition

RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing

Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies

CmnRec: Sequential Recommendations with Chunk-accelerated Memory Network

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictions

AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning

Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning

X-PEFT: eXtremely Parameter-Efficient Fine-Tuning for Extreme Multi-Profile Scenarios