USE: Dynamic User Modeling with Stateful Sequence Models

Zhihan Zhou,Qixiang Fang,Leonardo Neves,Francesco Barbieri,Yozen Liu,Han Liu,Maarten W. Bos,Ron Dotsch

2024-03-20

Abstract:User embeddings play a crucial role in user engagement forecasting and personalized services. Recent advances in sequence modeling have sparked interest in learning user embeddings from behavioral data. Yet behavior-based user embedding learning faces the unique challenge of dynamic user modeling. As users continuously interact with the apps, user embeddings should be periodically updated to account for users' recent and long-term behavior patterns. Existing methods highly rely on stateless sequence models that lack memory of historical behavior. They have to either discard historical data and use only the most recent data or reprocess the old and new data jointly. Both cases incur substantial computational overhead. To address this limitation, we introduce User Stateful Embedding (USE). USE generates user embeddings and reflects users' evolving behaviors without the need for exhaustive reprocessing by storing previous model states and revisiting them in the future. Furthermore, we introduce a novel training objective named future W-behavior prediction to transcend the limitations of next-token prediction by forecasting a broader horizon of upcoming user behaviors. By combining it with the Same User Prediction, a contrastive learning-based objective that predicts whether different segments of behavior sequences belong to the same user, we further improve the embeddings' distinctiveness and representativeness. We conducted experiments on 8 downstream tasks using Snapchat users' behavioral logs in both static (i.e., fixed user behavior sequences) and dynamic (i.e., periodically updated user behavior sequences) settings. We demonstrate USE's superior performance over established baselines. The results underscore USE's effectiveness and efficiency in integrating historical and recent user behavior sequences into user embeddings in dynamic user modeling.

Social and Information Networks,Artificial Intelligence,Computation and Language,Human-Computer Interaction,Information Retrieval,Machine Learning

What problem does this paper attempt to address?

The paper aims to address the challenges of user embeddings in dynamic user modeling, particularly how to efficiently update user embeddings as behavioral data continuously evolves while retaining historical behavior information.

Specifically, existing methods rely on stateless sequence models (such as Transformer), which either ignore historical data when processing new behavioral data, leading to the loss of long-term behavior patterns, or reprocess all data, resulting in significant computational overhead. To overcome this limitation, the authors propose UserStateful Embedding (USE), a method capable of storing and utilizing users' historical states to generate user embeddings without the need to reprocess all data. Additionally, they introduce a new training objective—Future W-Behavior Prediction (FBP), which aims to predict multiple possible future behaviors of users rather than being limited to the next behavior prediction. Combined with a contrastive learning method (Same User Prediction, SUP), it enhances the uniqueness and representativeness of the embedding representations.

Through experiments on actual behavior logs of Snapchat users, the authors demonstrate that USE outperforms existing baseline models in both static and dynamic settings and proves its effectiveness and efficiency in integrating historical and recent user behavior data. This work not only advances the research in user modeling but also sets new standards for personalized services in dynamic online environments.