Abstract:Relevance modeling is a critical component for enhancing user experience in search engines, with the primary objective of identifying items that align with users' queries. Traditional models only rely on the semantic congruence between queries and items to ascertain relevance. However, this approach represents merely one aspect of the relevance judgement, and is insufficient in isolation. Even powerful Large Language Models (LLMs) still cannot accurately judge the relevance of a query and an item from a semantic perspective. To augment LLMs-driven relevance modeling, this study proposes leveraging user interactions recorded in search logs to yield insights into users' implicit search intentions. The challenge lies in the effective prompting of LLMs to capture dynamic search intentions, which poses several obstacles in real-world relevance scenarios, i.e., the absence of domain-specific knowledge, the inadequacy of an isolated prompt, and the prohibitive costs associated with deploying LLMs. In response, we propose ProRBP, a novel Progressive Retrieved Behavior-augmented Prompting framework for integrating search scenario-oriented knowledge with LLMs effectively. Specifically, we perform the user-driven behavior neighbors retrieval from the daily search logs to obtain domain-specific knowledge in time, retrieving candidates that users consider to meet their expectations. Then, we guide LLMs for relevance modeling by employing advanced prompting techniques that progressively improve the outputs of the LLMs, followed by a progressive aggregation with comprehensive consideration of diverse aspects. For online serving, we have developed an industrial application framework tailored for the deployment of LLMs in relevance modeling. Experiments on real-world industry data and online A/B testing demonstrate our proposal achieves promising performance.

Exploring Query Auto-Completion And Click Logs For Contextual-Aware Web Search And Query Suggestion

Improving Multi-Faceted Book Search by Incorporating Sparse Latent Semantic Analysis of Click-Through Logs

Context-aware Query Suggestion by Mining Click-Through and Session Data

A Context-Aware Click Model for Web Search.

A Vlhmm Approach to Context-Aware Search

Mining Concept Sequences from Large-Scale Search Logs for Context-Aware Query Suggestion

Learning Query Ambiguity Models by Using Search Logs

Towards context-aware search by learning a very large variable length hidden markov model from search logs.

Query Expansion by Mining User Logs

Towards Boosting LLMs-driven Relevance Modeling with Progressive Retrieved Behavior-augmented Prompting

User Behavior Modeling for Better Web Search Ranking

Improving Entity Recommendation with Search Log and Multi-Task Learning

Query-oriented Data Augmentation for Session Search

A Graph-Enhanced Click Model for Web Search

Probabilistic Query Expansion Using Query Logs

Learning to Generate Personalized Query Auto-Completions Via a Multi-View Multi-Task Attentive Approach

Query Expansion for Short Queries by Mining User Logs

Leverage Implicit Feedback for Context-aware Product Search

A new query expansion method based on query logs mining.

Incorporating Non-sequential interactions into Click Models

Sogou Query Log Analysis: A Case Study for Collaborative Recommendation or Personalized IR