On Causally Disentangled State Representation Learning for Reinforcement Learning based Recommender Systems

Siyu Wang,Xiaocong Chen,Lina Yao

2024-07-18

Abstract:In Reinforcement Learning-based Recommender Systems (RLRS), the complexity and dynamism of user interactions often result in high-dimensional and noisy state spaces, making it challenging to discern which aspects of the state are truly influential in driving the decision-making process. This issue is exacerbated by the evolving nature of user preferences and behaviors, requiring the recommender system to adaptively focus on the most relevant information for decision-making while preserving generaliability. To tackle this problem, we introduce an innovative causal approach for decomposing the state and extracting \textbf{C}ausal-\textbf{I}n\textbf{D}ispensable \textbf{S}tate Representations (CIDS) in RLRS. Our method concentrates on identifying the \textbf{D}irectly \textbf{A}ction-\textbf{I}nfluenced \textbf{S}tate Variables (DAIS) and \textbf{A}ction-\textbf{I}nfluence \textbf{A}ncestors (AIA), which are essential for making effective recommendations. By leveraging conditional mutual information, we develop a framework that not only discerns the causal relationships within the generative process but also isolates critical state variables from the typically dense and high-dimensional state representations. We provide theoretical evidence for the identifiability of these variables. Then, by making use of the identified causal relationship, we construct causal-indispensable state representations, enabling the training of policies over a more advantageous subset of the agent's state space. We demonstrate the efficacy of our approach through extensive experiments, showcasing our method outperforms state-of-the-art methods.

Artificial Intelligence,Information Retrieval

What problem does this paper attempt to address?

This paper attempts to address the issues of high-dimensional state space and noise in Reinforcement Learning-based Recommender Systems (RLRS) caused by the complexity and dynamism of user interactions. These issues make it difficult to identify which aspects of the state truly impact the decision-making process, especially in the context of constantly changing user preferences and behaviors. Therefore, the paper proposes an innovative causal method to decompose the state and extract Causally Indispensable State Representations (CIDS) to improve the decision-making effectiveness of recommender systems.

Specifically, the paper focuses on identifying Directly Action-Influenced State variables (DAIS) and Action-Influenced Ancestors (AIA). By leveraging conditional mutual information, the paper develops a framework that not only identifies causal relationships in the generative process but also isolates key state variables from typically dense and high-dimensional state representations. Additionally, the paper provides theoretical evidence of the identifiability of these variables and demonstrates the effectiveness of its method through experiments, proving that the method outperforms existing state-of-the-art approaches.