Aligning Large Language Models for Controllable Recommendations

Wensheng Lu,Jianxun Lian,Wei Zhang,Guanghua Li,Mingyang Zhou,Hao Liao,Xing Xie

2024-08-04

Abstract:Inspired by the exceptional general intelligence of Large Language Models (LLMs), researchers have begun to explore their application in pioneering the next generation of recommender systems - systems that are conversational, explainable, and controllable. However, existing literature primarily concentrates on integrating domain-specific knowledge into LLMs to enhance accuracy, often neglecting the ability to follow instructions. To address this gap, we initially introduce a collection of supervised learning tasks, augmented with labels derived from a conventional recommender model, aimed at explicitly improving LLMs' proficiency in adhering to recommendation-specific instructions. Subsequently, we develop a reinforcement learning-based alignment procedure to further strengthen LLMs' aptitude in responding to users' intentions and mitigating formatting errors. Through extensive experiments on two real-world datasets, our method markedly advances the capability of LLMs to comply with instructions within recommender systems, while sustaining a high level of accuracy performance.

Information Retrieval,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the application issues of large language models (LLMs) in recommendation systems, particularly the challenges in controllability, interpretability, and conversational ability. Specifically, the goals of the paper include:

1. **Enhancing Control Ability**: Improve the ability of LLMs to follow recommendation-related instructions, enabling them to generate recommendation results based on specific user requirements.
2. **Reducing Format Errors**: Address common format error issues in LLMs when generating recommendation lists, such as duplicates or items that the user has already interacted with.
3. **Improving Generality and Adaptability**: Through supervised learning (SL) and reinforcement learning (RL) phases, enable LLMs to not only improve recommendation accuracy but also better understand and execute diverse recommendation instructions.

The proposed method achieves these goals through two stages: first, by injecting domain knowledge through a series of fine-tuning tasks and enhancing labels via a teacher model; second, by further optimizing the model's instruction-following ability using reinforcement learning. Experimental results show that this method significantly improves the performance of LLMs in real-world recommendation systems, particularly in following complex instructions and reducing format errors.