Abstract:User modeling, which aims to capture users' characteristics or interests, heavily relies on task-specific labeled data and suffers from the data sparsity issue. Several recent studies tackled this problem by pre-training the user model on massive user behavior sequences with a contrastive learning task. Generally, these methods assume different views of the same behavior sequence constructed via data augmentation are semantically consistent, i.e., reflecting similar characteristics or interests of the user, and thus maximizing their agreement in the feature space. However, due to the diverse interests and heavy noise in user behaviors, existing augmentation methods tend to lose certain characteristics of the user or introduce noisy behaviors. Thus, forcing the user model to directly maximize the similarity between the augmented views may result in a negative transfer. To this end, we propose to replace the contrastive learning task with a new pretext task: Augmentation-Adaptive SelfSupervised Ranking (AdaptSSR), which alleviates the requirement of semantic consistency between the augmented views while pre-training a discriminative user model. Specifically, we adopt a multiple pairwise ranking loss which trains the user model to capture the similarity orders between the implicitly augmented view, the explicitly augmented view, and views from other users. We further employ an in-batch hard negative sampling strategy to facilitate model training. Moreover, considering the distinct impacts of data augmentation on different behavior sequences, we design an augmentation-adaptive fusion mechanism to automatically adjust the similarity order constraint applied to each sample based on the estimated similarity between the augmented views. Extensive experiments on both public and industrial datasets with six downstream tasks verify the effectiveness of AdaptSSR.

UserBERT: Pre-training User Model with Contrastive Self-supervision

UserBERT: Contrastive User Model Pre-training

PTUM: Pre-training User Model from Unlabeled User Behaviors Via Self-supervision.

UserBERT: Modeling Long- and Short-Term User Preferences via Self-Supervision

Learning Transferable User Representations with Sequential Behaviors Via Contrastive Pre-training.

AdaptSSR: Pre-training User Model with Augmentation-Adaptive Self-Supervised Ranking

U-BERT: Pre-training User Representations for Improved Recommendation

URLBERT:A Contrastive and Adversarial Pre-trained Model for URL Classification

MCP: Self-supervised Pre-training for Personalized Chatbots with Multi-level Contrastive Sampling

Pre-Training with Whole Word Masking for Chinese BERT

W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training

Contrastive Pre-training for Deep Session Data Understanding

Interest-oriented Universal User Representation via Contrastive Learning

DialogueBERT: A Self-Supervised Learning based Dialogue Pre-training Encoder

HuBERTopic: Enhancing Semantic Representation of HuBERT through Self-supervision Utilizing Topic Model

Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis

Contrastive Curriculum Learning for Sequential User Behavior Modeling via Data Augmentation

Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems

SignBERT+: Hand-Model-Aware Self-Supervised Pre-Training for Sign Language Understanding.

Robust User Behavioral Sequence Representation via Multi-scale Stochastic Distribution Prediction

Learning Interest-oriented Universal User Representation Via Self-supervision