Abstract:Dense Passage Retrieval (DPR) serves as a crucial initial step in improving the performance of the Retrieval Augmented Generation paradigm for large language models. While DPRs are challenging to train and typically involve fine-tuning (FT) on pre-trained models to enhance embedding similarity between queries and associated textual data, the utilization of low-rank adaptation (LoRA) in DPR remains limited. A more comprehensive understanding of DPR fine-tuning is necessary to fully leverage its potential. The goal of this research is to refine DPR through the use of parameter-efficient fine-tuning (PEFT)'s LoRA, which is used to stabilize the pre-trained model weights and introduces trainable rank decomposition matrices into each layer of the transformer architecture. This approach has the potential to reduce the training parameters and deployment costs of the FT-based approach. The results of our experiments show that LoRA-DPR, when trained for three epochs on a Quadro RTX 8000 48GB GPU, requires only 14.32 million parameters for fine-tuning (11.64% of full parameter fine-tuning) and outperforms full parameter fine-tuning by 0.14 percentage points on the MRR@10 of the MS-MARCO dev set. The results show that this approach performs comparably to or even outperforms full FT models on Natural Questions and MS-MARCO datasets, addressing challenges related to resource deployment costs and lengthy training times. The source code is available at https://github.com/lauren-ai/LoRA_DPR.

Lora for dense passage retrieval of ConTextual masked auto-encoding