Abstract:Large language models (LLMs) face significant challenges stemming from their inherent limitations in knowledge, memory, alignment, and action. These challenges cannot be addressed by LLMs alone, but should rely on assistance from the external world, such as knowledge base, memory store, demonstration examples, and tools. Retrieval augmentation stands as a vital mechanism for bridging the gap between LLMs and the external assistance. However, conventional methods encounter two pressing issues. On the one hand, the general-purpose retrievers are not properly optimized for the retrieval augmentation of LLMs. On the other hand, the task-specific retrievers lack the required versatility, hindering their performance across the diverse retrieval augmentation scenarios. In this work, we present a novel approach, the LLM-Embedder, which comprehensively supports the diverse retrieval augmentation needs of LLMs with one unified embedding model. Training such a unified model is non-trivial, as various retrieval tasks aim to capture distinct semantic relationships, often subject to mutual interference. To address this challenge, we systematically optimize our training methodology. This includes reward formulation based on LLMs' feedback, the stabilization of knowledge distillation, multi-task fine-tuning with explicit instructions, and homogeneous in-batch negative sampling. These optimization strategies contribute to the outstanding empirical performance of the LLM-Embedder. Notably, it yields remarkable enhancements in retrieval augmentation for LLMs, surpassing both general-purpose and task-specific retrievers in various evaluation scenarios. Our checkpoint and source code are publicly available at <a class="link-external link-https" href="https://github.com/FlagOpen/FlagEmbedding" rel="external noopener nofollow">this https URL</a>.

LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models

EmbedLLM: Learning Compact Representations of Large Language Models

Language Models are Universal Embedders

ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning

Lens: Rethinking Multilingual Enhancement for Large Language Models

User-LLM: Efficient LLM Contextualization with User Embeddings

InfMLLM: A Unified Framework for Visual-Language Tasks.

Mufu: Multilingual Fused Learning for Low-Resource Translation with LLM

How Vocabulary Sharing Facilitates Multilingualism in LLaMA?

Investigating Decoder-only Large Language Models for Speech-to-text Translation

LLMs are Also Effective Embedding Models: An In-depth Overview

Retrieve Anything To Augment Large Language Models

LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

eFedLLM: Efficient LLM Inference Based on Federated Learning

Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems

Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings

LuxEmbedder: A Cross-Lingual Approach to Enhanced Luxembourgish Sentence Embeddings

LLM for Everyone: Representing the Underrepresented in Large Language Models

LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding

Large Language Models for Expansion of Spoken Language Understanding Systems to New Languages

Multi-Lingual Malaysian Embedding: Leveraging Large Language Models for Semantic Representations