Abstract:Decoder-only large language model (LLM)-based embedding models are beginning to outperform BERT or T5-based embedding models in general-purpose text embedding tasks, including dense vector-based retrieval. In this work, we introduce the NV-Embed model with a variety of architectural designs and training procedures to significantly enhance the performance of LLM as a versatile embedding model, while maintaining its simplicity and reproducibility. For model architecture, we propose a latent attention layer to obtain pooled embeddings, which consistently improves retrieval and downstream task accuracy compared to mean pooling or using the last <EOS> token embedding from LLMs. To enhance representation learning, we remove the causal attention mask of LLMs during contrastive training. For model training, we introduce a two-stage contrastive instruction-tuning method. It first applies contrastive training with instructions on retrieval datasets, utilizing in-batch negatives and curated hard negative examples. At stage-2, it blends various non-retrieval datasets into instruction tuning, which not only enhances non-retrieval task accuracy but also improves retrieval performance. Combining these techniques, our NV-Embed model, using only publicly available data, has achieved a record-high score of 69.32, ranking No. 1 on the Massive Text Embedding Benchmark (MTEB) (as of May 24, 2024), with 56 tasks, encompassing retrieval, reranking, classification, clustering, and semantic textual similarity tasks. Notably, our model also attains the highest score of 59.36 on 15 retrieval tasks in the MTEB benchmark (also known as BEIR). We will open-source the model at: <a class="link-external link-https" href="https://huggingface.co/nvidia/NV-Embed-v1" rel="external noopener nofollow">this https URL</a>.

Nomic Embed: Training a Reproducible Long Context Text Embedder

Nomic Embed Vision: Expanding the Latent Space

LongEmbed: Extending Embedding Models for Long Context Retrieval

Vec2Vec: A Compact Neural Network Approach for Transforming Text Embeddings with High Fidelity

Multilingual E5 Text Embeddings: A Technical Report

Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models

YaRN: Efficient Context Window Extension of Large Language Models

Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents

NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

NewsEmbed: Modeling News through Pre-trained Document Representations

Nugget: Neural Agglomerative Embeddings of Text

Making Text Embedders Few-Shot Learners

Training With "Paraphrasing the Original Text'' Improves Long-Context Performance

ACER: Automatic Language Model Context Extension via Retrieval

ChuXin: 1.6B Technical Report

Training-Free Long-Context Scaling of Large Language Models

RULER: What's the Real Context Size of Your Long-Context Language Models?

Recurrent Context Compression: Efficiently Expanding the Context Window of LLM

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context.