Abstract:User embeddings play a crucial role in user engagement forecasting and personalized services. Recent advances in sequence modeling have sparked interest in learning user embeddings from behavioral data. Yet behavior-based user embedding learning faces the unique challenge of dynamic user modeling. As users continuously interact with the apps, user embeddings should be periodically updated to account for users' recent and long-term behavior patterns. Existing methods highly rely on stateless sequence models that lack memory of historical behavior. They have to either discard historical data and use only the most recent data or reprocess the old and new data jointly. Both cases incur substantial computational overhead. To address this limitation, we introduce User Stateful Embedding (USE). USE generates user embeddings and reflects users' evolving behaviors without the need for exhaustive reprocessing by storing previous model states and revisiting them in the future. Furthermore, we introduce a novel training objective named future W-behavior prediction to transcend the limitations of next-token prediction by forecasting a broader horizon of upcoming user behaviors. By combining it with the Same User Prediction, a contrastive learning-based objective that predicts whether different segments of behavior sequences belong to the same user, we further improve the embeddings' distinctiveness and representativeness. We conducted experiments on 8 downstream tasks using Snapchat users' behavioral logs in both static (i.e., fixed user behavior sequences) and dynamic (i.e., periodically updated user behavior sequences) settings. We demonstrate USE's superior performance over established baselines. The results underscore USE's effectiveness and efficiency in integrating historical and recent user behavior sequences into user embeddings in dynamic user modeling.

Dynamic Embeddings for User Profiling in Twitter

Weakly Supervised User Profile Extraction from Twitter.

Cross-Network Social User Embedding with Hybrid Differential Privacy Guarantees

Dynamic Network Embeddings for Network Evolution Analysis.

Fighting Redundancy and Model Decay with Embeddings

Embedding and Predicting the Event at Early Stage

Exploiting Structural and Temporal Influence for Dynamic Social-Aware Recommendation

Cross-media User Profiling with Joint Textual and Social User Embedding.

Temporal dynamics of user activities: deep learning strategies and mathematical modeling for long-term and short-term profiling

Event Early Embedding: Predicting Event Volume Dynamics at Early Stage

D-Sempre: Learning Deep Semantic-Preserving Embeddings for User interests-Social Contents Modeling

Social User Profiling: A Social-Aware Topic Modeling Perspective

Dynamic Network Embedding : an Extended Approach for Skip-gram Based Network Embedding

Dynamic Word Embeddings for Evolving Semantic Discovery

Real-time Filtering on Interest Profiles in Twitter Stream

Collaborative Dynamic Sparse Topic Regression with User Profile Evolution for Item Recommendation

USE: Dynamic User Modeling with Stateful Sequence Models

HE-SNE: Heterogeneous Event Sequence-based Streaming Network Embedding for Dynamic Behaviors

A User Adaptive Model for Followee Recommendation on Twitter

Adaptive Evolutionary Filtering in Real-Time Twitter Stream

User Profile Preserving Social Network Embedding