Survey for Landing Generative AI in Social and E-commerce Recsys -- the Industry Perspectives

Da Xu,Danqing Zhang,Guangyu Yang,Bo Yang,Shuyuan Xu,Lingling Zheng,Cindy Liang

2024-06-11

Abstract:Recently, generative AI (GAI), with their emerging capabilities, have presented unique opportunities for augmenting and revolutionizing industrial recommender systems (Recsys). Despite growing research efforts at the intersection of these fields, the integration of GAI into industrial Recsys remains in its infancy, largely due to the intricate nature of modern industrial Recsys infrastructure, operations, and product sophistication. Drawing upon our experiences in successfully integrating GAI into several major social and e-commerce platforms, this survey aims to comprehensively examine the underlying system and AI foundations, solution frameworks, connections to key research advancements, as well as summarize the practical insights and challenges encountered in the endeavor to integrate GAI into industrial Recsys. As pioneering work in this domain, we hope outline the representative developments of relevant fields, shed lights on practical GAI adoptions in the industry, and motivate future research.

Information Retrieval,Artificial Intelligence

What problem does this paper attempt to address?

This paper is a survey from an industry perspective on the application of Generative Artificial Intelligence (GAI) in social and e-commerce recommendation systems (Recsys). The main objective of the paper is to address how to effectively integrate and apply GAI in practical industrial Recsys, which is a challenge due to the complexity, operational procedures, and product complexity of modern Recsys.

The paper first outlines the key deficiencies of existing Recsys, such as the lack of open-world memory and generalization, the lack of proactive content planning, and interactive reasoning capabilities. Then, the paper discusses the foundations of GAI, including the operation and management of large-scale language models (LLMOps), and how to enhance personalized recommendations, content planning, and model design through GAI.

The authors propose a classification framework to organize the applications of GAI in Recsys based on actual tasks and categories of GAI methods, and emphasize the progress of using GAI in data processing, feature engineering, and model building. They also discuss solution frameworks and practical considerations, including how to integrate GAI into the development process of Recsys, and how to address the challenges of evaluation, deployment, and iteration.

In addition, the paper discusses how to enhance the content planning capability of Recsys through GAI, introducing Retrieval-Augmented Generation (RAG) technology, which utilizes external knowledge to reposition, explain, and plan content to meet the diverse needs of users and improve transparency and trustworthiness.

Finally, the paper proposes introducing autonomous AI agents to facilitate interactive recommendations and feedback loops. These agents can engage in reasoning, use tools, and handle users' free-form inputs to improve user experience and surpass the capabilities of current dialogic Recsys. The paper also addresses the responsible AI and human-AI alignment issues when deploying GAI in customer-oriented applications.

In conclusion, this paper aims to summarize the current application status of GAI in social and e-commerce Recsys, provide practical experiences, identify challenges, and provide directions for future research.