Abstract:Semantic identifier (ID) is an important concept in information retrieval that aims to preserve the semantics of objects such as documents and items inside their IDs. Previous studies typically adopt a two-stage pipeline to learn semantic IDs by first procuring embeddings using off-the-shelf text encoders and then deriving IDs based on the embeddings. However, each step introduces potential information loss, and there is usually an inherent mismatch between the distribution of embeddings within the latent space produced by text encoders and the anticipated distribution required for semantic indexing. It is non-trivial to design a method that can learn the document's semantic representations and its hierarchical structure simultaneously, given that semantic IDs are discrete and sequentially structured, and the semantic supervision is deficient. In this paper, we introduce LMIndexer, a self-supervised framework to learn semantic IDs with a generative language model. We tackle the challenge of sequential discrete ID by introducing a semantic indexer capable of generating neural sequential discrete representations with progressive training and contrastive learning. In response to the semantic supervision deficiency, we propose to train the model with a self-supervised document reconstruction objective. We show the high quality of the learned IDs and demonstrate their effectiveness on three tasks including recommendation, product search, and document retrieval on five datasets from various domains. Code is available at <a class="link-external link-https" href="https://github.com/PeterGriffinJin/LMIndexer" rel="external noopener nofollow">this https URL</a>.

LSM: Language Sense Model for Information Retrieval

Do Multi-Sense Embeddings Improve Natural Language Understanding?

A Unified Model for Word Sense Representation and Disambiguation.

Using Term Sense to Improve Language Modeling Approach to Genomic IR

A Simplified Latent Semantic Indexing Approach for Multi-Linguistic Information Retrieval.

TDSS: A New Word Sense Representation Framework for Information Retrieval

Text Information Retrieval Based on Latent Semantic Analysis

SMM,A Sense Matrix Model

DocReLM: Mastering Document Retrieval with Language Model

Statistical Word Sense Aware Topic Models

Adapting Learned Sparse Retrieval for Long Documents

LATENT SEMANTIC INDEXING (LSI) AND ITS APPLICATIONS IN CHINESE TEXT PROCESSING

Using Word Sense As a Latent Variable in LDA Can Improve Topic Modeling.

Topic Models Incorporating Statistical Word Senses

Self-Retrieval: Building an Information Retrieval System with One Large Language Model

MINERS: Multilingual Language Models as Semantic Retrievers

Language Models As Semantic Indexers

LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding

Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints

Multi-document Chinese Name Disambiguation Based on Latent Semantic Analysis

Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation