On Relevance in Information Retrieval Based on Relevance Theory

CHENG Ying,SUN Jian-jun,LI Bao-qiang

2010-01-01

Abstract:The relevance assessment module of information retrieval interaction model is annotated with information processing model of relevance theory.The conclusion of this paper is that it is feasible to annotate relevance assessment of information retrieval with information processing model of relevance theory.

What problem does this paper attempt to address?

On Relevance in Information Retrieval Based on Relevance Theory——information production,indexing

成颖,孙建军,李宝强

2010-01-01

Abstract:Based on the work of Saracevic and Harter,the authors argue that relevance theory can become the basic theory of relevance research in information retrieval.The two modules of information retrieval interaction model,information production and indexing,are annotated with relevance theory.
A Linguistic Study on Relevance Modeling in Information Retrieval

Yixing Fan,Jiafeng Guo,Xinyu Ma,Ruqing Zhang,Yanyan Lan,Xueqi Cheng

DOI: https://doi.org/10.1145/3442381.3450009

2021-03-01

Abstract:Relevance plays a central role in information retrieval (IR), which has received extensive studies starting from the 20th century. The definition and the modeling of relevance has always been critical challenges in both information science and computer science research areas. Along with the debate and exploration on relevance, IR has already become a core task in many real-world applications, such as Web search engines, question answering systems, conversational bots, and so on. While relevance acts as a unified concept in all these retrieval tasks, the inherent definitions are quite different due to the heterogeneity of these tasks. This raises a question to us: Do these different forms of relevance really lead to different modeling focuses? To answer this question, in this work, we conduct an empirical study on relevance modeling in three representative IR tasks, i.e., document retrieval, answer retrieval, and response retrieval. Specifically, we attempt to study the following two questions: 1) Does relevance modeling in these tasks really show differences in terms of natural language understanding (NLU)? We employ 16 linguistic tasks to probe a unified retrieval model over these three retrieval tasks to answer this question. 2) If there do exist differences, how can we leverage the findings to enhance the relevance modeling? We proposed three intervention methods to investigate how to leverage different modeling focuses of relevance to improve these IR tasks. We believe the way we study the problem as well as our findings would be beneficial to the IR community.

Information Retrieval
When does Relevance Mean Usefulness and User Satisfaction in Web Search?

Jiaxin Mao,Yiqun Liu,Ke Zhou,Jian-Yun Nie,Jingtao Song,Min Zhang,Shaoping Ma,Jiashen Sun,Hengliang Luo

DOI: https://doi.org/10.1145/2911451.2911507

2016-01-01

Abstract:Relevance is a fundamental concept in information retrieval (IR) studies. It is however often observed that relevance as annotated by secondary assessors may not necessarily mean usefulness and satisfaction perceived by users. In this study, we confirm the difference by a laboratory study in which we collect relevance annotations by external assessors, usefulness and user satisfaction information by users, for a set of search tasks. We also find that a measure based on usefulness rather than relevance annotated has a better correlation with user satisfaction. However, we show that external assessors are capable of annotating usefulness when provided with more search context information. In addition, we also show that it is possible to generate automatically usefulness labels when some training data is available. Our findings explain why traditional system-centric evaluation metrics are not well aligned with user satisfaction and suggest that a usefulness-based evaluation method can be defined to better reflect the quality of search systems perceived by the users.
A Relevance Evaluation Model of Information Retrieval Based on Weighted Term Frequency

ZHANG Chun-qing,CHEN Chao,SHAO Zheng-rong,YU Neng-hai

DOI: https://doi.org/10.3969/j.issn.1006-9348.2008.01.035

2008-01-01

Abstract:Relevance evaluation model is an important research issue in the field of information retrieval.The bas- ic information retrieval models are boolean model,vector space model and probabilistic model.The latter two models are implemented in many retrieval systems extensively but the different position of query term in every document is ig- nored.Some researches have considered the information HTML tags but the scheme of assigning weighted parameters is not ideal.In this paper,WTFM(Weighted Term Frequency Model)is proposed according to the existence of term frequency(TF).And these weighted coefficients are learned by simulated annealing algorithm.The results of the ex- periments show that the introduction of TF's weights brings improvements to the system.
User-Oriented Relevance Judgment: A Conceptual Model

Zhiwei Chen,Yunjie Xu

DOI: https://doi.org/10.1109/HICSS.2005.654

2005-01-01

Abstract:The concept of relevance has been heatedly debated in last decade. Not satisfied with the narrow and technical definition of system relevance, researchers turn to the subjective and situational aspect of this concept. How does a user perceive a document as relevant? The literature on relevance has identified numerous factors affecting such judgment. Taking a cognitive approach, this study focuses on the criteria users employ in making relevance judgment. Based on Grice's theory of communication, this paper proposes a five-factor model of relevance: topicality, novelty, reliability, understandability, and scope. Data are collected from a semi-controlled survey study and analyzed following a psychometric procedure. The result supports topicality and novelty as the key relevance criteria. Theoretical and practical implications of this study are discussed.
A Statistical Model for Relevance Feedback in Information Retrieval

C. T. Yu,W. S. Luk,T. Y. Cheung

DOI: https://doi.org/10.1145/321941.321947

1976-01-01

Abstract:A statistical model is presented for the investigation of a practical method used in relevance feedback. A necessary and sufficient condition for the two parameters used in this method to define a better query than the original query is given. A region in the plane of the parameters is shown to satisfy the sufficient condition. While the points for producing optimal queries are not exactly located, they are shown to be lying on a finite portion of a hyperbola. Experimental results support some of the theoretical findings.
Understanding Relevance Judgments in Legal Case Retrieval

Yunqiu Shao,Yueyue Wu,Yiqun Liu,Jiaxin Mao,Shaoping Ma

DOI: https://doi.org/10.1145/3569929

IF: 4.657

2023-02-07

ACM Transactions on Information Systems

Abstract:Legal case retrieval, which aims to retrieve relevant cases given a query case, has drawn increasing research attention in recent years. While much research has worked on developing automatic retrieval models, how to characterize relevance in this specialized information retrieval (IR) task is still an open question. Towards an in-depth understanding of relevance judgments, we conduct a laboratory user study that involves 72 participants of different domain expertise. In the user study, we collect the relevance score along with detailed explanations for the relevance judgment and various measures of the judgment process. From the collected data, we observe that both the subjective (e.g., domain expertise) and objective (e.g., query/case property) factors influence the relevance judgment process. By investigating the collected user explanations, we identify task-specific patterns of user attention distribution and re-think the criteria for relevance judgments. Moreover, we investigate the similarity in attention distribution between models and users. Further, we propose a two-stage framework that utilizes user attention to improve relevance estimation for legal case retrieval. Our study sheds light on understanding relevance judgments in legal case retrieval and provides implications for improving the design of corresponding retrieval systems.

computer science, information systems
Relevance Estimation with Multiple Information Sources on Search Engine Result Pages.

Junqi Zhang,Yiqun Liu,Shaoping Ma,Qi Tian

DOI: https://doi.org/10.1145/3269206.3271673

2018-01-01

Abstract:Relevance estimation is among the most important tasks in the ranking of search results because most search engines follow the Probability Ranking Principle. Current relevance estimation methodologies mainly concentrate on text matching between the query and Web documents, link analysis and user behavior models. However, users judge the relevance of search results directly from Search Engine Result Pages (SERPs), which provide valuable signals for reranking. Morden search engines aggregate heterogeneous information items (such as images, news, and hyperlinks) to a single ranking list on SERPs. The aggregated search results have different visual patterns, textual semantics and presentation structures, and a better strategy should rely on all these information sources to improve ranking performance. In this paper, we propose a novel framework named Joint Relevance Estimation model (JRE), which learns the visual patterns from screenshots of search results, explores the presentation structures from HTML source codes and also adopts the semantic information of textual contents. To evaluate the performance of the proposed model, we construct a large scale practical Search Result Relevance (SRR) dataset which consists of multiple information sources and 4-grade relevance scores of over 60,000 search results. Experimental results show that the proposed JRE model achieves better performance than state-of-the-art ranking solutions as well as the original ranking of commercial search engines.
Relevance Judgment in Epistemic and Hedonic Information Searches

Yunjie Xu

DOI: https://doi.org/10.1002/asi.20461

2006-01-01

Journal of the American Society for Information Science and Technology

Abstract:Research in information science now regards users' relevance judgment as subjective perception. However, user-centered studies in the extant literature mainly focus on relevance judgment in problem solving contexts in which the situational relevance of a document is the main concern for users. This study investigates users' relevance judgment in non-problem-solving contexts, i.e., when users search information for epistemic value or entertainment. It is posited that informative relevance and affective relevance should be the main concerns for users. Based on H. P. Grice's (1975, 1989) communication theory and Y. Xu and Z. Chen's (2006) framework, this study tests the significance of topicality, novelty, reliability, understandability, and scope to informative relevance and affective relevance in non-problem-solving contexts. This empirical study finds novelty, reliability, and topicality to be key aspects of informative relevance. © 2007 Wiley Periodicals, Inc.
Relevance Judgment: What Do Information Users Consider Beyond Topicality?

YJ Xu,ZW Chen

DOI: https://doi.org/10.1002/asi.20361

2006-01-01

Journal of the American Society for Information Science and Technology

Abstract:How does an information user perceive a document as relevant? The literature on relevance has identified numerous factors affecting such a judgment. Taking a cognitive approach, this study focuses ...
Research of Relevance Feedback Technology in Personalized Information Retrieval

Song Weiping,Yang Jianlin

2008-01-01

Abstract:Facing with the efficient, accurate and personalized information requirements of present users, this paper atroduces the inpplication of relevance feedback technology in personalized information retrieval. Based the research status and basic idea of relevance feedback firstly, and after analyzing the methods of relevance feedback realization and the differences on the design between different systems, it proposes a in tegrated model of relevance feedback and personalized information retrieval.At last, the paper discusses the application of data fusion idea in relevance feedback in order to optimize feedback results and improve the effectiveness of retrieval system’s performance.
On Models of Information Retrieval Processes

CT YU,WS LUK,MK SIU

DOI: https://doi.org/10.1016/0306-4379(79)90004-8

IF: 3.18

1979-01-01

Information Systems

Abstract:A model called the binary independence model is presented as a generalization of a few models which have applied to the analyses of a clustered search process, some indexing strategies and a relevance feedback process. This model, together with the Swets model, the linked-2-Poisson model, the 2-Poisson model, the binary limited-dependence model, the tree dependence model, the binary dependence model and the non-binary independent model are compared and contrasted. Despite the fact that these models are intended for different applications, three aspects of modelling are identified, based on which, these models are compared. The three aspects are the class concept, the distribution of similarities and the relation between a matching function and its retrieval effectiveness. As results of the comparison between the models, more insight into the models is gained and a set of guidelines are suggested to help a system designer to choose a model.
Investigating Passage-level Relevance and Its Role in Document-level Relevance Judgment

Zhijing Wu,Jiaxin Mao,Yiqun Liu,Min Zhang,Shaoping Ma

DOI: https://doi.org/10.1145/3331184.3331233

2019-01-01

Abstract:The understanding of the process of relevance judgment helps to inspire the design of retrieval models. Traditional retrieval models usually estimate relevance based on document-level signals. Recent works consider a more fine-grain, passage-level relevance information, which can further enhance retrieval performance. However, it lacks a detailed analysis of how passage-level relevance signals determine or influence the relevance judgment of the whole document. To investigate the role of passage-level relevance in the document-level relevance judgment, we construct an ad-hoc retrieval dataset with both passage-level and document-level relevance labels. A thorough analysis reveals that: 1) there is a strong correlation between the document-level relevance and the fractions of irrelevant passages to highly relevant passages; 2) the position, length and query similarity of passages play different roles in the determination of document-level relevance; 3) The sequential passage-level relevance within a document is a potential indicator for the document-level relevance. Based on the relationship between passage-level and document-level relevance, we also show that utilizing passage-level relevance signals can improve existing document ranking models. This study helps us better understand how users perceive relevance for a document and inspire the designing of novel ranking models leveraging fine-grain, passage-level relevance signals.
Understanding the Role of Human-Inspired Heuristics for Retrieval Models

Li Xiangsheng,Liu Yiqun,Mao Jiaxin

DOI: https://doi.org/10.1007/s11704-020-0016-y

IF: 2.6688

2021-01-01

Frontiers of Computer Science

Abstract:Relevance estimation is one of the core concerns of information retrieval (IR) studies. Although existing retrieval models gained much success in both deepening our understanding of information seeking behavior and building effective retrieval systems, we have to admit that the models work in a rather different manner from how humans make relevance judgments. Users’ information seeking behaviors involve complex cognitive processes, however, the majority of these behavior patterns are not considered in existing retrieval models. To bridge the gap between practical user behavior and retrieval model, it is essential to systematically investigate user cognitive behavior during relevance judgement and incorporate these heuristics into retrieval models. In this paper, we aim to formally define a set of basic user reading heuristics during relevance judgement and investigate their corresponding modeling strategies in retrieval models. Further experiments are conducted to evaluate the effectiveness of different reading heuristics for improving ranking performance. Based on a large-scale Web search dataset, we find that most reading heuristics can improve the performance of retrieval model and establish guidelines for improving the design of retrieval models with humaninspired heuristics. Our study sheds light on building retrieval model from the perspective of cognitive behavior.
Fully utilize feedbacks: language model based relevance feedback in information retrieval

Sheng-Long Lv,Zhi-Hong Deng,Hang Yu,Ning Gao,Jia-Jian Jiang

DOI: https://doi.org/10.1007/978-3-642-25853-4_30

2011-01-01

Abstract:Relevance feedback algorithm is proposed to be an effective way to improve the precision of information retrieval. However, most researches about relevance feedback are based on vector space model, which can't be used in other more complicated and powerful models, such as language model and logic model. Meanwhile, other researches are conceptually restricted to the view of a query as a set of terms, and so cannot be naturally applied to more general case when the query is considered as a sequence of terms and the frequency information of a query tern is considered. In this paper, we mainly focuses on relevant feedback Algorithm based on language model. We use a mixture model to describe the process of generating document and use EM to solve model's parameters. Our research also employs semi-supervised learning to calculate collection model and proposes an effective way to obtain feedback from irrelevant documents to improve our algorithm.
Empirical Analysis on Relevance Criteria Oriented Academic Information Retrieval System Success Model

Cheng Ying

DOI: https://doi.org/10.11925/infotech.1003-3513.2011.10.09

2011-01-01

Abstract:In this paper, empirical analysis on AIRSSM is carried out based on structural equation model. The results show that special emphasis of scholarly information retrieval systems analysis and design should be focused on: ( 1 ) System quality and information quality oriented; (2) A balanced consideration of all factors affecting system quality; (3) Authority, integrity, real-time and effectiveness are essential to information quality.
A Unified Relevance Model for Opinion Retrieval

Xuanjing Huang,W. Bruce Croft

DOI: https://doi.org/10.1145/1645953.1646075

2009-01-01

Abstract:Representing the information need is the greatest challenge for opinion retrieval. Typical queries for opinion retrieval are composed of either just content words, or content words with a small number of cue "opinion" words. Both are inadequate for retrieving opinionated documents. In this paper, we develop a general formal framework--the opinion relevance model--to represent an information need for opinion retrieval. We explore a series of methods to automatically identify the most appropriate opinion words for query expansion, including using query independent sentiment resources. We also propose a relevance feedback-based approach to extract opinion words. Both query-independent and query-dependent methods can also be integrated into a more effective mixture relevance model. Finally, opinion retrieval experiments are presented for the Blog06 and COAE08 text collections. The results show that, significant improvements can always be obtained by this opinion relevance model whether sentiment resources are available or not.
Modeling Diverse Relevance Patterns in Ad-hoc Retrieval

Yixing Fan,Jiafeng Guo,Yanyan Lan,Jun Xu,Chengxiang Zhai,Xueqi Cheng

DOI: https://doi.org/10.1145/3209978.3209980

2018-01-01

Abstract:Assessing relevance between a query and a document is challenging in ad-hoc retrieval due to its diverse patterns, i.e., a document could be relevant to a query as a whole or partially as long as it provides sufficient information for users' need. Such diverse relevance patterns require an ideal retrieval model to be able to assess relevance in the right granularity adaptively. Unfortunately, most existing retrieval models compute relevance at a single granularity, either document-wide or passage-level, or use fixed combination strategy, restricting their ability in capturing diverse relevance patterns. In this work, we propose a data-driven method to allow relevance signals at different granularities to compete with each other for final relevance assessment. Specifically, we propose a HIerarchical Neural maTching model (HiNT) which consists of two stacked components, namely local matching layer and global decision layer. The local matching layer focuses on producing a set of local relevance signals by modeling the semantic matching between a query and each passage of a document. The global decision layer accumulates local signals into different granularities and allows them to compete with each other to decide the final relevance score. Experimental results demonstrate that our HiNT model outperforms existing state-of-the-art retrieval models significantly on benchmark ad-hoc retrieval datasets.
Search Result Reranking with Visual and Structure Information Sources

Yiqun Liu,Junqi Zhang,Jiaxin Mao,Zhang Min,Shaoping Ma,Qi Tian,Yanxiong Lu,Leyu Lin

DOI: https://doi.org/10.1145/3329188

2019-01-01

Abstract:Relevance estimation is among the most important tasks in the ranking of search results. Current methodologies mainly concentrate on text matching, link analysis, and user behavior models. However, users judge the relevance of search results directly from Search Engine Result Pages (SERPs), which provide valuable signals for reranking. In this article, we propose two different approaches to aggregate the visual, structure, as well as textual information sources of search results in relevance estimation. The first one is a late-fusion framework named Joint Relevance Estimation model (JRE). JRE estimates the relevance independently from screenshots, textual contents, and HTML source codes of search results and jointly makes the final decision through an inter-modality attention mechanism. The second one is an early-fusion framework named Tree-based Deep Neural Network (TreeNN), which embeds the texts and images into the HTML parse tree through a recursive process. To evaluate the performance of the proposed models, we construct a large-scale practical Search Result Relevance (SRR) dataset that consists of multiple information sources and relevance labels of over 60,000 search results. Experimental results show that the proposed two models achieve better performance than state-of-the-art ranking solutions as well as the original rankings of commercial search engines.
Modelling Dynamic Interactions between Relevance Dimensions

Sagar Uprety,Shahram Dehdashti,Lauren Fell,Peter Bruza,Dawei Song

DOI: https://doi.org/10.48550/arXiv.1907.10943

2019-07-25

Abstract:Relevance is an underlying concept in the field of Information Science and Retrieval. It is a cognitive notion consisting of several different criteria or dimensions. Theoretical models of relevance allude to interdependence between these dimensions, where their interaction and fusion leads to the final inference of relevance. We study the interaction between the relevance dimensions using the mathematical framework of Quantum Theory. It is considered a generalised framework to model decision making under uncertainty, involving multiple perspectives and influenced by context. Specifically, we conduct a user study by constructing the cognitive analogue of a famous experiment in Quantum Physics. The data is used to construct a complex-valued vector space model of the user's cognitive state, which is used to explain incompatibility and interference between relevance dimensions. The implications of our findings to inform the design of Information Retrieval systems are also discussed.

Information Retrieval

On Relevance in Information Retrieval Based on Relevance Theory

On Relevance in Information Retrieval Based on Relevance Theory——information production,indexing

A Linguistic Study on Relevance Modeling in Information Retrieval

When does Relevance Mean Usefulness and User Satisfaction in Web Search?

A Relevance Evaluation Model of Information Retrieval Based on Weighted Term Frequency

User-Oriented Relevance Judgment: A Conceptual Model

A Statistical Model for Relevance Feedback in Information Retrieval

Understanding Relevance Judgments in Legal Case Retrieval

Relevance Estimation with Multiple Information Sources on Search Engine Result Pages.

Relevance Judgment in Epistemic and Hedonic Information Searches

Relevance Judgment: What Do Information Users Consider Beyond Topicality?

Research of Relevance Feedback Technology in Personalized Information Retrieval

On Models of Information Retrieval Processes

Investigating Passage-level Relevance and Its Role in Document-level Relevance Judgment

Understanding the Role of Human-Inspired Heuristics for Retrieval Models

Fully utilize feedbacks: language model based relevance feedback in information retrieval

Empirical Analysis on Relevance Criteria Oriented Academic Information Retrieval System Success Model

A Unified Relevance Model for Opinion Retrieval

Modeling Diverse Relevance Patterns in Ad-hoc Retrieval

Search Result Reranking with Visual and Structure Information Sources

Modelling Dynamic Interactions between Relevance Dimensions