Abstract:Sequential recommendation models user interests based on historical behaviors to provide personalized recommendation. Previous sequential recommendation algorithms primarily employ neural networks to extract features of user interests, achieving good performance. However, due to the recommendation system datasets sparsity, these algorithms often employ small-scale network frameworks, resulting in weaker generalization capability. Recently, a series of sequential recommendation algorithms based on large pre-trained language models have been proposed. Nonetheless, given the real-time demands of recommendation systems, the challenge remains in applying pre-trained language models for rapid recommendations in real scenarios. To address this, we propose a sequential recommendation algorithm based on a pre-trained language model and knowledge distillation. The key of proposed algorithm is to transfer pre-trained knowledge across domains and achieve lightweight inference by knowledge distillation. The algorithm operates in two stages: in the first stage, we fine-tune the pre-trained language model on the recommendation dataset to transfer the pre-trained knowledge to the recommendation task; in the second stage, we distill the trained language model to transfer the learned knowledge to a lightweight model. Extensive experiments on multiple public recommendation datasets show that the proposed algorithm enhances recommendation accuracy and provide timely recommendation services.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the contradiction between data sparsity and real - time requirements in the recommendation system. Specifically, although the existing sequential recommendation algorithms can extract user - interest features through neural network models, due to the sparsity of the recommendation data set, these models usually adopt small - scale network frameworks, resulting in weak generalization ability. Although some sequential recommendation algorithms based on large - scale pre - trained language models (Pre - trained Language Model, PLM) have been proposed recently, and these algorithms have improved in recommendation accuracy, since the recommendation system needs to provide rapid recommendations when users browse web pages, watch videos or shop online, how to apply pre - trained language models in actual recommendation scenarios to achieve rapid recommendations is still a great challenge. To solve this problem, the paper proposes a lightweight sequential recommendation algorithm that combines pre - trained language models and knowledge distillation. The core idea of this algorithm is to transfer the knowledge obtained by the pre - trained model from a large - scale corpus across domains to the recommendation scenario, and refine and extract this knowledge into a small - scale model through knowledge distillation, thereby achieving rapid recommendations while improving the recommendation effect. The specific steps include: 1. **Stage 1**: Use the pre - trained language model to perform fine - tuning training on the recommendation data set, and transfer the pre - trained knowledge of the model across domains to the recommendation task. 2. **Stage 2**: Perform knowledge distillation on the trained language model, and transfer the learned knowledge to a lightweight model to achieve lightweight inference. Through these two stages, the paper aims to solve the contradiction between data sparsity and real - time requirements in the recommendation system, and improve the performance and response speed of the recommendation system.

Pre-trained Language Model and Knowledge Distillation for Lightweight Sequential Recommendation

Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Model

Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Models

Multi-stage Knowledge Distillation for Sequential Recommendation with Interest Knowledge

Ensemble Modeling with Contrastive Knowledge Distillation for Sequential Recommendation

One Model for All: Large Language Models are Domain-Agnostic Recommendation Systems

SLMRec: Empowering Small Language Models for Sequential Recommendation

Rethinking Large Language Model Architectures for Sequential Recommendations

Knowledge Prompt-tuning for Sequential Recommendation

Denoising Pre-Training and Customized Prompt Learning for Efficient Multi-Behavior Sequential Recommendation

KERL: A Knowledge-Guided Reinforcement Learning Model for Sequential Recommendation

Leave No One Behind: Online Self-Supervised Self-Distillation for Sequential Recommendation

Adversarial Distillation for Efficient Recommendation with External Knowledge

Knowledge Distillation Approaches for Accurate and Efficient Recommender System

Curriculum-scheduled Knowledge Distillation from Multiple Pre-trained Teachers for Multi-domain Sequential Recommendation

Can Small Language Models be Good Reasoners for Sequential Recommendation?

Knowledge Distillation Application Technology for Chinese NLP

LlamaRec: Two-Stage Recommendation using Large Language Models for Ranking

Parameters Efficient Fine-Tuning for Long-Tailed Sequential Recommendation.

Pre-train, Align, and Disentangle: Empowering Sequential Recommendation with Large Language Models

Enhancing Sequential Recommenders with Augmented Knowledge from Aligned Large Language Models