Abstract:Ranking has always been one of the top concerns in information retrieval researches. For decades, the lexical matching signal has dominated the ad-hoc retrieval process, but solely using this signal in retrieval may cause the vocabulary mismatch problem. In recent years, with the development of representation learning techniques, many researchers turn to Dense Retrieval (DR) models for better ranking performance. Although several existing DR models have already obtained promising results, their performance improvement heavily relies on the sampling of training examples. Many effective sampling strategies are not efficient enough for practical usage, and for most of them, there still lacks theoretical analysis in how and why performance improvement happens. To shed light on these research questions, we theoretically investigate different training strategies for DR models and try to explain why hard negative sampling performs better than random sampling. Through the analysis, we also find that there are many potential risks in static hard negative sampling, which is employed by many existing training methods. Therefore, we propose two training strategies named a Stable Training Algorithm for dense Retrieval (STAR) and a query-side training Algorithm for Directly Optimizing Ranking pErformance (ADORE), respectively. STAR improves the stability of DR training process by introducing random negatives. ADORE replaces the widely-adopted static hard negative sampling method with a dynamic one to directly optimize the ranking performance. Experimental results on two publicly available retrieval benchmark datasets show that either strategy gains significant improvements over existing competitive baselines and a combination of them leads to the best performance.

Mitigating the Impact of False Negatives in Dense Retrieval with Contrastive Confidence Regularization

Noisy Pair Corrector for Dense Retrieval

Unsupervised dense retrieval with conterfactual contrastive learning

Hard Negatives or False Negatives: Correcting Pooling Bias in Training Neural Ranking Models

Optimizing Dense Retrieval Model Training with Hard Negatives.

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval

Contrastive Refinement for Dense Retrieval Inference in the Open-Domain Question Answering Task

Early-Learning Regularized Contrastive Learning for Cross-Modal Retrieval with Noisy Labels

Noise-Robust Dense Retrieval via Contrastive Alignment Post Training

A Multiple Positives Enhanced NCE Loss for Image-Text Retrieval

Multi-views Contrastive Learning for Dense Text Retrieval

Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions

Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training

Simple contrastive learning in a self-supervised manner for robust visual question answering

Jo-SRC: A Contrastive Approach for Combating Noisy Labels

One-bit Active Query with Contrastive Pairs

SyNeg: LLM-Driven Synthetic Hard-Negatives for Dense Retrieval

Deep Evidential Learning with Noisy Correspondence for Cross-modal Retrieval

Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning

Curriculum Contrastive Context Denoising for Few-shot Conversational Dense Retrieval

Learning with Noisy Correspondence