Abstract:Leveraging generative retrieval (GR) techniques to enhance search systems is an emerging methodology that has shown promising results in recent years. In GR, a text-to-text model maps string queries directly to relevant document identifiers (docIDs), dramatically simplifying the retrieval process. However, when applying most GR models in large-scale E-commerce for personalized item search, we must face two key problems in encoding and decoding. (1) Existing docID generation methods ignore the encoding of efficiency information, which is critical in E-commerce. (2) The positional information is important in decoding docIDs, while prior studies have not adequately discriminated the significance of positional information or well exploited the inherent interrelation among these positions. To overcome these problems, we introduce an efficient Hierarchical encoding-decoding Generative retrieval method (Hi-Gen) for large-scale personalized E-commerce search systems. Specifically, we first design a representation learning model using metric learning to learn discriminative feature representations of items to capture semantic relevance and efficiency information. Then, we propose a category-guided hierarchical clustering scheme that makes full use of the semantic and efficiency information of items to facilitate docID generation. Finally, we design a position-aware loss to discriminate the importance of positions and mine the inherent interrelation between different tokens at the same position. This loss boosts the performance of the language model used in the decoding stage. Besides, we propose two variants of Hi-Gen (Hi-Gen-I2I and Hi-Gen-Cluster) to support online real-time large-scale recall in the online serving process. Hi-Gen gets 3.30% and 4.62% improvements over SOTA for Recall@1 on the public and industry datasets, respectively.

An Efficient Corpus Indexer for dynamic corpora retrieval

Continual Learning for Generative Retrieval over Dynamic Corpora

Exploring the Practicality of Generative Retrieval on Dynamic Corpora

A Neural Corpus Indexer for Document Retrieval

Hi-Gen: Generative Retrieval For Large-Scale Personalized E-commerce Search

Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval

Generative Dense Retrieval: Memory Can Be a Burden

Auto Search Indexer for End-to-End Document Retrieval

CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks

CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks

ACE: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling

A Distributed Collaborative Retrieval Framework Excelling in All Queries and Corpora based on Zero-shot Rank-Oriented Automatic Evaluation

GLEN: Generative Retrieval via Lexical Index Learning

A Survey of Generative Information Retrieval

EHI: End-to-end Learning of Hierarchical Index for Efficient Dense Retrieval

Generative Retrieval with Few-shot Indexing

Generative Retrieval with Preference Optimization for E-commerce Search

From Matching to Generation: A Survey on Generative Information Retrieval

Generative Retrieval Meets Multi-Graded Relevance

CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks