Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking

Shengyao Zhuang,Bing Liu,Bevan Koopman,Guido Zuccon

2023-10-20

Abstract:In the field of information retrieval, Query Likelihood Models (QLMs) rank documents based on the probability of generating the query given the content of a document. Recently, advanced large language models (LLMs) have emerged as effective QLMs, showcasing promising ranking capabilities. This paper focuses on investigating the genuine zero-shot ranking effectiveness of recent LLMs, which are solely pre-trained on unstructured text data without supervised instruction fine-tuning. Our findings reveal the robust zero-shot ranking ability of such LLMs, highlighting that additional instruction fine-tuning may hinder effectiveness unless a question generation task is present in the fine-tuning dataset. Furthermore, we introduce a novel state-of-the-art ranking system that integrates LLM-based QLMs with a hybrid zero-shot retriever, demonstrating exceptional effectiveness in both zero-shot and few-shot scenarios. We make our codebase publicly available at <a class="link-external link-https" href="https://github.com/ielab/llm-qlm" rel="external noopener nofollow">this https URL</a>.

Information Retrieval,Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is in the field of information retrieval, how to utilize large - language models (LLMs) to achieve strong zero - sample query likelihood model (QLMs) document ranking capabilities without additional fine - tuning data. Specifically, the paper focuses on the real - ranking effects of recent LLMs in zero - sample settings. These models are pre - trained only with unsupervised text data and without instruction fine - tuning. The research finds that such LLMs have strong zero - sample ranking capabilities, and additional instruction fine - tuning may reduce their effectiveness, unless the fine - tuning data set contains the task of generating questions.

To verify this, the paper introduces a new state - of - the - art ranking system, which combines LLM - based QLMs with a hybrid zero - sample retriever and performs well in both zero - sample and few - sample scenarios. In addition, the paper also explores the effects of different popular instruction - fine - tuned LLMs in zero - sample ranking tasks, and finds that further instruction fine - tuning will damage the effectiveness of QLM ranking, especially when the fine - tuning data set lacks specific tasks of generating questions.

In conclusion, the paper aims to explore and prove the strong ranking capabilities of LLMs without additional fine - tuning in zero - sample query likelihood models, and proposes a new method that combines zero - sample retrievers and LLM - based QLMs to improve the overall performance of information retrieval systems.