Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Model

Yu Cui,Feng Liu,Pengbo Wang,Bohao Wang,Heng Tang,Yi Wan,Jun Wang,Jiawei Chen

DOI: https://doi.org/10.1145/3640457.3688118

2024-08-20

Abstract:Owing to their powerful semantic reasoning capabilities, Large Language Models (LLMs) have been effectively utilized as recommenders, achieving impressive performance. However, the high inference latency of LLMs significantly restricts their practical deployment. To address this issue, this work investigates knowledge distillation from cumbersome LLM-based recommendation models to lightweight conventional sequential models. It encounters three challenges: 1) the teacher's knowledge may not always be reliable; 2) the capacity gap between the teacher and student makes it difficult for the student to assimilate the teacher's knowledge; 3) divergence in semantic space poses a challenge to distill the knowledge from embeddings. To tackle these challenges, this work proposes a novel distillation strategy, DLLM2Rec, specifically tailored for knowledge distillation from LLM-based recommendation models to conventional sequential models. DLLM2Rec comprises: 1) Importance-aware ranking distillation, which filters reliable and student-friendly knowledge by weighting instances according to teacher confidence and student-teacher consistency; 2) Collaborative embedding distillation integrates knowledge from teacher embeddings with collaborative signals mined from the data. Extensive experiments demonstrate the effectiveness of the proposed DLLM2Rec, boosting three typical sequential models with an average improvement of 47.97%, even enabling them to surpass LLM-based recommenders in some cases.

Information Retrieval

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

The paper aims to address the inference efficiency issue of large language models (LLMs) in recommendation systems. Specifically, although LLMs perform excellently in recommendation tasks due to their powerful semantic reasoning capabilities, their high inference latency limits practical deployment. To solve this problem, the paper proposes a new knowledge distillation strategy—**DLLM2Rec**, which can distill the knowledge of complex LLM recommendation models into lightweight traditional sequential recommendation models.

#### Main Challenges
1. **Teacher Knowledge Reliability**: The knowledge of LLMs may be unreliable, especially in some cases where it is even inferior to traditional models.
2. **Model Capacity Gap**: The significant capacity difference between LLMs and traditional models makes it difficult for the student model to fully absorb the teacher's knowledge.
3. **Semantic Space Discrepancy**: LLMs model users and items based on content, while traditional models extract collaborative signals based on user interaction behavior, leading to poor alignment of their embedding spaces.

#### Solutions
- **Importance-Aware Ranking Distillation**: By introducing importance weights to filter reliable and student-friendly instances for distillation.
- **Collaborative Embedding Distillation**: By using learnable projectors to map the teacher model's embeddings to the student model's embedding space and combining collaborative signals to generate enhanced student embeddings.

Through these methods, DLLM2Rec can significantly improve the performance of traditional recommendation models without sacrificing inference efficiency, even surpassing LLM recommendation models in some cases.