Abstract:Learning low-dimensional representation for large number of products present in an e-commerce catalogue plays a vital role as they are helpful in tasks like product ranking, product recommendation, finding similar products, modelling user-behaviour etc. Recently, a lot of tasks in the NLP field are getting tackled using the Transformer based models and these deep models are widely applicable in the industries setting to solve various problems. With this motivation, we apply transformer based model for learning contextual representation of products in an e-commerce setting. In this work, we propose a novel approach of pre-training transformer based model on a users generated sessions dataset obtained from a large fashion e-commerce platform to obtain latent product representation. Once pre-trained, we show that the low-dimension representation of the products can be obtained given the product attributes information as a textual sentence. We mainly pre-train BERT, RoBERTa, ALBERT and XLNET variants of transformer model and show a quantitative analysis of the products representation obtained from these models with respect to Next Product Recommendation(NPR) and Content Ranking(CR) tasks. For both the tasks, we collect an evaluation data from the fashion e-commerce platform and observe that XLNET model outperform other variants with a MRR of 0.5 for NPR and NDCG of 0.634 for CR. XLNET model also outperforms the Word2Vec based non-transformer baseline on both the downstream tasks. To the best of our knowledge, this is the first and novel work for pre-training transformer based models using users generated sessions data containing products that are represented with rich attributes information for adoption in e-commerce setting. These models can be further fine-tuned in order to solve various downstream tasks in e-commerce, thereby eliminating the need to train a model from scratch.

BERT2DNN: BERT Distillation with Massive Unlabeled Data for Online E-Commerce Search

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains.

Explainable LLM-driven Multi-dimensional Distillation for E-Commerce Relevance Learning

ReprBERT: Distilling BERT to an Efficient Representation-Based Relevance Model for E-Commerce

Best Practices for Distilling Large Language Models into BERT for Web Search Ranking

Knowledge Distillation based Contextual Relevance Matching for E-commerce Product Search

Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation

Towards Better Seach Query Classification with Distribution-Diverse Multi-Expert Knowledge Distillation in JD Ads Search

TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval

Pre-training Tasks for User Intent Detection and Embedding Retrieval in E-commerce Search

Fine-tune BERT for E-commerce Non-Default Search Ranking

Distilling Knowledge from BERT into Simple Fully Connected Neural Networks for Efficient Vertical Retrieval.

Performance and sustainability of BERT derivatives in dyadic data

UserBERT: Modeling Long- and Short-Term User Preferences via Self-Supervision

ListBERT: Learning to Rank E-commerce products with Listwise BERT

Deep Bag-of-Words Model: An Efficient and Interpretable Relevance Architecture for Chinese E-Commerce

Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation

XtremeDistil: Multi-stage Distillation for Massive Multilingual Models

Cross-domain Attention Network with Wasserstein Regularizers for E-commerce Search

Learning-To-Embed: Adopting Transformer based models for E-commerce Products Representation Learning

BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distance