Abstract:Pre-trained language models like BERT have achieved great success in a wide variety of NLP tasks, while the superior performance comes with high demand in computational resources, which hinders the application in low-latency IR systems. We present TwinBERT model for effective and efficient retrieval, which has twin-structured BERT-like encoders to represent query and document respectively and a crossing layer to combine the embeddings and produce a similarity score. Different from BERT, where the two input sentences are concatenated and encoded together, TwinBERT decouples them during encoding and produces the embeddings for query and document independently, which allows document embeddings to be pre-computed offline and cached in memory. Thereupon, the computation left for run-time is from the query encoding and query-document crossing only. This single change can save large amount of computation time and resources, and therefore significantly improve serving efficiency. Moreover, a few well-designed network layers and training strategies are proposed to further reduce computational cost while at the same time keep the performance as remarkable as BERT model. Lastly, we develop two versions of TwinBERT for retrieval and relevance tasks correspondingly, and both of them achieve close or on-par performance to BERT-Base model. The model was trained following the teacher-student framework and evaluated with data from one of the major search engines. Experimental results showed that the inference time was significantly reduced and was firstly controlled around 20ms on CPUs while at the same time the performance gain from fine-tuned BERT-Base model was mostly retained. Integration of the models into production systems also demonstrated remarkable improvements on relevance metrics with negligible influence on latency.

ERDL: Efficient Retrieval Framework Based on Distillation from Large Language Models

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains.

Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment

TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval

A Multi-level Distillation based Dense Passage Retrieval Model

Discriminative Language Model via Self-Teaching for Dense Retrieval

Building an Efficient and Effective Retrieval-based Dialogue System via Mutual Learning

ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder Via Self On-the-fly Distillation for Dense Passage Retrieval

Pre-trained Language Model for Web-scale Retrieval in Baidu Search

QuadrupletBERT: an Efficient Model for Embedding-Based Large-Scale Retrieval

UnifieR: A Unified Retriever for Large-Scale Retrieval

ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval

LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval.

Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently

Self-Retrieval: Building an Information Retrieval System with One Large Language Model

Making Large Language Models A Better Foundation For Dense Retrieval

Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiency

Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy

RETA-LLM: A Retrieval-Augmented Large Language Model Toolkit

XtremeDistil: Multi-stage Distillation for Massive Multilingual Models

Distillation Enhanced Generative Retrieval