Practice on Effectively Extracting NLP Features for Click-Through Rate Prediction

Hao Yang,Ziliang Wang,Weijie Bian,Yifan Zeng
DOI: https://doi.org/10.1145/3583780.3614707
2023-10-21
Abstract:Click-through rate (CTR) prediction is critical for industrial applications such as recommendation system and online advertising. Practically, there are a series of research proved that Natural Language Processing (NLP) features are helpful to improve CTR task performance. As these works show, there are different ways to extract NLP features. For example, keywords of item title are extracted as open-box feature by term frequency?inverse document frequency (tf-idf) method while item semantic embedding is extracted as black-box feature by shallow models (\emphe.g., word2vec) or deep learning models (e.g., BERT). However, these NLP models are pre-trained on NLP task, which is very different from the CTR task. Then it leads to the limited improvement of Area Under the ROC Curve (AUC) in CTR task. On the other hand, traditional NLP models for CTR task only consider open-box feature or black-box feature separately, which also leads to the discounted effect. Lastly, many NLP models are mainly used to extract semantic features only on item side. These methods take little account of user side information, or only IDs related features (\emphe.g., item's IDs) in user behavior sequence are introduced. In our work, we proposed a new network named BERT Attention method based on both Item and User information (BAIU). The target of BAIU is consistent with the CTR task, which is helpful to extract more effective NLP features. Also, both open-box and black-box features are simultaneously extracted by this network, which makes the model to learn more useful NLP features for CTR task and also makes the model more interpretable. Extensive experiments on both public data and our commercial data validate the effectiveness of our approach. Finally, the online experiment brings 2.2% gain of CTR on recommendation.
Computer Science
What problem does this paper attempt to address?