Tuning Query Reformulator with Fine-Grained Relevance Feedback
Yuchen Zhai,Yong Jiang,Yue Zhang,Jianhui Ji,Rong Xiao,Haitao Tang,Chen Li,Pengjun Xie,Yin Zhang
DOI: https://doi.org/10.1007/978-981-99-7596-9_15
2023-01-01
Abstract:Pseudo-relevance feedback (PRF) has been empirically validated as an effective query reformulation method to improve retrieval performance. Recent studies formulate query reformulation as a reinforcement learning task to directly optimize the retrieval performance. However, this paradigm computes the feedback signals by comparing the retrieved documents with the manual annotations, and neglects that the annotations severely suffer from the unlabeled problem (the relevant documents of a query may not be fully annotated), causing the model to overfit the training set. Moreover, the training of reinforcement learning is expensive and unstable. To address the above problems, inspired by recent great achievements of reinforcement learning from human feedback (RLHF), we propose a simple fine-grained feedback framework for query reformulation, which computes the feedback signals by a powerful re-ranking model instead of manual annotations. Specially, we first utilize various automation methods to generate annotated data, which allows us to initialize the reformulator and obtain a good starting point. Then we employ a re-ranking model to assign fine-grained scores to the rewritten queries generated by the reformulator. Finally, we refine the reformulator using feedback scores. In this way, the knowledge of the re-ranking model can be effectively transferred to the reformulator, leading to a better generalization performance. Furthermore, our framework can enhance performance by leveraging a large amount of unlabeled data. Experiments on a real-world E-Commerce search engine and three public benchmarks demonstrate the effectiveness of our framework.