Abstract:The aim of information retrieval is to efficiently retrieve the most relevant information based on user queries. With the rise of pre-trained language models (such as BERT, GPT, etc.), researchers have begun to utilize the dense vector representation capabilities of pre-trained language models, proposing dense retrieval methods to better capture semantic information. The emergence of large language models (such as ChatGPT) has prompted researchers to start exploring the application of these models in actual retrieval tasks. However, the introduction of large language models also brings about an increase in computational and storage costs. To address the issues brought about by large language models, this study proposes an Efficient Retrieval framework based on Distillation from Large language models(ERDL). This framework initially enhances the representational capacity of the encoder-only model by means of knowledge distillation from large language models, improving the accuracy and relevance of retrieval while maintaining the efficiency advantage of the encoder-only model (which is typically smaller in size). Then, it utilizes the encoding capabilities of the large language model to compensate for the missing information in the encoder-only model’s representation, further enhancing the performance of the encoder-only model through contrastive learning supervised by the large language model. Experimental results indicate that our method, compared to large language models, has achieved significant improvements in the three real-world datasets of the MTEB information retrieval task. While ensuring that the encoder-only model has competitive retrieval results, our method has improved the retrieval speed by over 85%, effectively reducing computational costs.

Distillation for Multilingual Information Retrieval

Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation

Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models

Learning Cross-Lingual IR from an English Retriever

Distilling Efficient Language-Specific Models for Cross-Lingual Transfer

Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation

Cross-lingual Information Retrieval with BERT

Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval

Exploiting Neural Query Translation into Cross Lingual Information Retrieval

HLTCOE at TREC 2023 NeuCLIR Track

Multilingual Multimodal Learning with Machine Translated Text

Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation

Enhancing Model Performance in Multilingual Information Retrieval with Comprehensive Data Engineering Techniques

Synthetic Cross-language Information Retrieval Training Data

Synergistic Approach for Simultaneous Optimization of Monolingual, Cross-lingual, and Multilingual Information Retrieval

Cross-Lingual Training with Dense Retrieval for Document Retrieval

Extending Translate-Train for ColBERT-X to African Language CLIR

ERDL: Efficient Retrieval Framework Based on Distillation from Large Language Models

Cross-Lingual Relevance Transfer for Document Retrieval

ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder Via Self On-the-fly Distillation for Dense Passage Retrieval

C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval