Abstract: Transformer encoding networks have been proved to be a powerful tool of understanding natural languages. They are playing a critical role in native ads service, which facilitates the recommendation of appropriate ads based on user's web browsing history. For the sake of efficient recommendation, conventional methods would generate user and advertisement embeddings independently with a siamese transformer encoder, such that approximate nearest neighbour search (ANN) can be leveraged. Given that the underlying semantic about user and ad can be complicated, such independently generated embeddings are prone to information loss, which leads to inferior recommendation quality. Although another encoding strategy, the cross encoder, can be much more accurate, it will lead to huge running cost and become infeasible for realtime services, like native ads recommendation. In this work, we propose hybrid encoder, which makes efficient and precise native ads recommendation through two consecutive steps: retrieval and ranking. In the retrieval step, user and ad are encoded with a siamese component, which enables relevant candidates to be retrieved via ANN search. In the ranking step, it further represents each ad with disentangled embeddings and each user with ad-related embeddings, which contributes to the fine-grained selection of high-quality ads from the candidate set. Both steps are light-weighted, thanks to the pre-computed and cached intermedia results. To optimize the hybrid encoder's performance in this two-stage workflow, a progressive training pipeline is developed, which builds up the model's capability in the retrieval and ranking task step-by-step. The hybrid encoder's effectiveness is experimentally verified: with very little additional cost, it outperforms the siamese encoder significantly and achieves comparable recommendation quality as the cross encoder.

Towards more effective encoders in pre-training for sequential recommendation

Contrastive Pre-training for Sequential Recommendation

Enhancing CTR Prediction through Sequential Recommendation Pre-training: Introducing the SRP4CTR Framework

UPRec: User-aware Pre-training for Sequential Recommendation

Integrating the Pre-trained Item Representations with Reformed Self-attention Network for Sequential Recommendation

BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer

An encoder enhanced sequence recommendation algorithm

Multimodal Pre-training Framework for Sequential Recommendation via Contrastive Learning

Self-Supervised Multi-Modal Sequential Recommendation

Pretraining Binarization Encoders for Recommendation Acceleration

Sequential Recommendation with a Pre-trained Module Learning Multi-modal Information

Hybrid Encoder: Towards Efficient and Precise Native AdsRecommendation via Hybrid Transformer Encoding Networks

Two-stage Sequential Recommendation Via Bidirectional Attentive Behavior Embedding and Long/Short-term Integration

Denoising Pre-Training and Customized Prompt Learning for Efficient Multi-Behavior Sequential Recommendation

Collaborative Sequential Recommendations via Multi-View GNN-Transformers

Large Multi-modal Encoders for Recommendation

Temporal Contrastive Pre-Training for Sequential Recommendation.

Multi-modality Meets Re-learning: Mitigating Negative Transfer in Sequential Recommendation

PRECISE: Pre-training Sequential Recommenders with Collaborative and Semantic Information

Contrastive Learning with Bidirectional Transformers for Sequential Recommendation

Long-Sequence Recommendation Models Need Decoupled Embeddings