Abstract:Ranker and retriever are two important components in dense passage retrieval. The retriever typically adopts a dual-encoder model, where queries and documents are separately input into two pre-trained models, and the vectors generated by the models are used for similarity calculation. The ranker often uses a cross-encoder model, where the concatenated query-document pairs are input into a pre-trained model to obtain word similarities. However, the dual-encoder model lacks interaction between queries and documents due to its independent encoding, while the cross-encoder model requires substantial computational cost for attention calculation, making it difficult to obtain real-time retrieval results. In this paper, we propose a dense retrieval model called MD2PR based on multi-level distillation. In this model, we distill the knowledge learned from the cross-encoder to the dual-encoder at both the sentence level and word level. Sentence-level distillation enhances the dual-encoder on capturing the themes and emotions of sentences. Word-level distillation improves the dual-encoder in analysis of word semantics and relationships. As a result, the dual-encoder can be used independently for subsequent encoding and retrieval, avoiding the significant computational cost associated with the participation of the cross-encoder. Furthermore, we propose a simple dynamic filtering method, which updates the threshold during multiple training iterations to ensure the effective identification of false negatives and thus obtains a more comprehensive semantic representation space. The experimental results over two standard datasets show our MD2PR outperforms 11 baseline models in terms of MRR and Recall metrics.

Multi-channel Reverse Dictionary Model.

RDMTL: Reverse dictionary model based on multitask learning

Multi-View Analysis Dictionary Learning for Image Classification.

A Unified Model for Reverse Dictionary and Definition Modelling

BERT for Monolingual and Cross-Lingual Reverse Dictionary.

A prompt construction method for the reverse dictionary task of large-scale language models

WordAlchemy: A transformer-based Reverse Dictionary

Reverse Modeling in Large Language Models

WantWords: an Open-source Online Reverse Dictionary System

Using Multi-Sense Vector Embeddings for Reverse Dictionaries

Multi-Task Learning for Multiple Language Translation.

MLR: A Two-stage Conversational Query Rewriting Model with Multi-task Learning

Deep Unfolding Convolutional Dictionary Model for Multi-Contrast MRI Super-resolution and Reconstruction

Multi-Domain Neural Machine Translation with Word-Level Domain Context Discrimination

Weakly-paired Deep Dictionary Learning for Cross-Modal Retrieval

A Multi-level Distillation based Dense Passage Retrieval Model

Multi-task Recurrent Model for True Multilingual Speech Recognition

Towards a Transformer-Based Reverse Dictionary Model for Quality Estimation of Definitions

Modeling Local Dependence in Natural Language with Multi-channel Recurrent Neural Networks

Top-Push Constrained Modality-Adaptive Dictionary Learning for Cross-Modality Person Re-Identification

Discriminative Dictionary Learning with Common Label Alignment for Cross-Modal Retrieval.