Abstract:Search engines are crucial as they provide an efficient and easy way to access vast amounts of information on the internet for diverse information needs. User queries, even with a specific need, can differ significantly. Prior research has explored the resilience of ranking models against typical query variations like paraphrasing, misspellings, and order changes. Yet, these works overlook how diverse demographics uniquely formulate identical queries. For instance, older individuals tend to construct queries more naturally and in varied order compared to other groups. This demographic diversity necessitates enhancing the adaptability of ranking models to diverse query formulations. To this end, in this paper, we propose a framework that integrates a novel rewriting pipeline that rewrites queries from various demographic perspectives and a novel framework to enhance ranking robustness. To be specific, we use Chain of Thought (CoT) technology to utilize Large Language Models (LLMs) as agents to emulate various demographic profiles, then use them for efficient query rewriting, and we innovate a robust Multi-gate Mixture of Experts (MMoE) architecture coupled with a hybrid loss function, collectively strengthening the ranking models' robustness. Our extensive experimentation on both public and industrial datasets assesses the efficacy of our query rewriting approach and the enhanced accuracy and robustness of the ranking model. The findings highlight the sophistication and effectiveness of our proposed model.
What problem does this paper attempt to address?
### What problems does this paper attempt to solve?
This paper aims to solve the **ranking model robustness problem** of search engines when facing queries from different user groups. Specifically, the paper focuses on the following points:
1. **Diverse query expressions**:
- Even if user queries have the same semantic requirements, they can show significant differences due to different personal backgrounds and expression habits. For example, the elderly tend to use more natural and variably - ordered query methods, while students are more likely to use keyword - driven queries.
- This diversity poses challenges to ranking models because they need to adapt to various query forms.
2. **Limitations of existing research**:
- Existing research mainly focuses on dealing with a single type of query variation (such as spelling mistakes, synonym replacement, etc.), but ignores the complex situation of mixing multiple query variations.
- Most research lacks attention to how different groups of people uniquely construct the same query, resulting in ranking models being not robust enough in practical applications.
3. **Stability of ranking models**:
- Ideally, a robust search engine should return consistent results for semantically consistent queries. However, existing ranking models often have unstable results when facing semantically similar but differently expressed queries, which affects the user experience.
### Solutions proposed in the paper
To solve the above problems, the paper proposes the following innovative methods:
1. **Multi - agent - based query rewriting framework**:
- Use large - language models (LLM) to simulate different population roles (such as women, men, students, the elderly), and rewrite queries from multiple semantic perspectives.
- Through the Chain of Thought (CoT) technology and a strict query verification process, ensure that the rewritten queries not only conform to the original intention but also to the expression style of the specified role.
2. **Architecture for enhancing ranking model robustness**:
- Introduce a new Robust Multi - gate Mixture of Experts (MMoE) architecture, which can dynamically identify the semantic commonalities between different rewritten queries, thereby improving the stability of ranking.
- Design a novel loss function, combined with Jensen - Shannon divergence to constrain the output distributions from different agent perspectives, while improving both accuracy and robustness.
### Formula representation
- The query rewriting process can be formalized as:
\[
F_{\theta_k}: Q\rightarrow\hat{Q}_k,\quad k\in[1,\ldots,K]
\]
where \(Q = \{q_0, q_1,\ldots, q_n\}\) is the original query set, and \(\hat{Q}_k=\{\hat{q}_{k,0},\hat{q}_{k,1},\ldots,\hat{q}_{k,n}\}\) is the rewritten query set from the \(k\) - th perspective.
- The evaluation formula for the ranking model is:
\[
E(q_c, D_c, Y_c)\sim M(\pi(q_c, D_c, R), Y_c)
\]
where \(M\) represents the evaluation metric, and \(\pi(q_c, D_c, R)\) represents the permutation generated by using the ranking model \(R\) for a given query \(q_c\) and document list \(D_c\).
- The robustness evaluation formula for the ranking model is:
\[
V(q_c, D_c)\sim M([\pi_k(q^k_c, D_c, R)]),\quad k\in[1,\ldots,K]
\]
where \(\pi_k(q^k_c, D_c, R)\) represents the ranking label list generated by using the ranking model \(R\) for the \(k\) - th rewritten query and its corresponding document.
Through these methods, the paper aims to improve the stability and accuracy of search engines when facing diverse queries, thereby improving the user experience.