Relevance Judgment: What Do Information Users Consider Beyond Topicality?

YJ Xu,ZW Chen

DOI: https://doi.org/10.1002/asi.20361

2006-01-01

Journal of the American Society for Information Science and Technology

Abstract:How does an information user perceive a document as relevant? The literature on relevance has identified numerous factors affecting such a judgment. Taking a cognitive approach, this study focuses ...

What problem does this paper attempt to address?

Relevance Judgment in Epistemic and Hedonic Information Searches

Yunjie Xu

DOI: https://doi.org/10.1002/asi.20461

2006-01-01

Journal of the American Society for Information Science and Technology

Abstract:Research in information science now regards users' relevance judgment as subjective perception. However, user-centered studies in the extant literature mainly focus on relevance judgment in problem solving contexts in which the situational relevance of a document is the main concern for users. This study investigates users' relevance judgment in non-problem-solving contexts, i.e., when users search information for epistemic value or entertainment. It is posited that informative relevance and affective relevance should be the main concerns for users. Based on H. P. Grice's (1975, 1989) communication theory and Y. Xu and Z. Chen's (2006) framework, this study tests the significance of topicality, novelty, reliability, understandability, and scope to informative relevance and affective relevance in non-problem-solving contexts. This empirical study finds novelty, reliability, and topicality to be key aspects of informative relevance. © 2007 Wiley Periodicals, Inc.
User-Oriented Relevance Judgment: A Conceptual Model

Zhiwei Chen,Yunjie Xu

DOI: https://doi.org/10.1109/HICSS.2005.654

2005-01-01

Abstract:The concept of relevance has been heatedly debated in last decade. Not satisfied with the narrow and technical definition of system relevance, researchers turn to the subjective and situational aspect of this concept. How does a user perceive a document as relevant? The literature on relevance has identified numerous factors affecting such judgment. Taking a cognitive approach, this study focuses on the criteria users employ in making relevance judgment. Based on Grice's theory of communication, this paper proposes a five-factor model of relevance: topicality, novelty, reliability, understandability, and scope. Data are collected from a semi-controlled survey study and analyzed following a psychometric procedure. The result supports topicality and novelty as the key relevance criteria. Theoretical and practical implications of this study are discussed.
Understanding Relevance Judgments in Legal Case Retrieval

Yunqiu Shao,Yueyue Wu,Yiqun Liu,Jiaxin Mao,Shaoping Ma

DOI: https://doi.org/10.1145/3569929

IF: 4.657

2023-02-07

ACM Transactions on Information Systems

Abstract:Legal case retrieval, which aims to retrieve relevant cases given a query case, has drawn increasing research attention in recent years. While much research has worked on developing automatic retrieval models, how to characterize relevance in this specialized information retrieval (IR) task is still an open question. Towards an in-depth understanding of relevance judgments, we conduct a laboratory user study that involves 72 participants of different domain expertise. In the user study, we collect the relevance score along with detailed explanations for the relevance judgment and various measures of the judgment process. From the collected data, we observe that both the subjective (e.g., domain expertise) and objective (e.g., query/case property) factors influence the relevance judgment process. By investigating the collected user explanations, we identify task-specific patterns of user attention distribution and re-think the criteria for relevance judgments. Moreover, we investigate the similarity in attention distribution between models and users. Further, we propose a two-stage framework that utilizes user attention to improve relevance estimation for legal case retrieval. Our study sheds light on understanding relevance judgments in legal case retrieval and provides implications for improving the design of corresponding retrieval systems.

computer science, information systems
Relevance judgments of mobile commercial information

Xiaolun Wang,Zhijuan Hong,Yunjie (Calvin) Xu,Chenghong Zhang,Hong Ling

DOI: https://doi.org/10.1002/asi.23060

2014-01-01

Abstract:AbstractIn the age of mobile commerce, users receive floods of commercial messages. How do users judge the relevance of such information? Is their relevance judgment affected by contextual factors, such as location and time? How do message content and contextual factors affect users' privacy concerns? With a focus on mobile ads, we propose a research model based on theories of relevance judgment and mobile marketing research. We suggest topicality, reliability, and economic value as key content factors and location and time as key contextual factors. We found mobile relevance judgment is affected mainly by content factors, whereas privacy concerns are affected by both content and contextual factors. Moreover, topicality and economic value have a synergetic effect that makes a message more relevant. Higher topicality and location precision exacerbate privacy concerns, whereas message reliability alleviates privacy concerns caused by location precision. These findings reveal an interesting intricacy in user relevance judgment and privacy concerns and provide nuanced guidance for the design and delivery of mobile commercial information.
Investigating Passage-level Relevance and Its Role in Document-level Relevance Judgment

Zhijing Wu,Jiaxin Mao,Yiqun Liu,Min Zhang,Shaoping Ma

DOI: https://doi.org/10.1145/3331184.3331233

2019-01-01

Abstract:The understanding of the process of relevance judgment helps to inspire the design of retrieval models. Traditional retrieval models usually estimate relevance based on document-level signals. Recent works consider a more fine-grain, passage-level relevance information, which can further enhance retrieval performance. However, it lacks a detailed analysis of how passage-level relevance signals determine or influence the relevance judgment of the whole document. To investigate the role of passage-level relevance in the document-level relevance judgment, we construct an ad-hoc retrieval dataset with both passage-level and document-level relevance labels. A thorough analysis reveals that: 1) there is a strong correlation between the document-level relevance and the fractions of irrelevant passages to highly relevant passages; 2) the position, length and query similarity of passages play different roles in the determination of document-level relevance; 3) The sequential passage-level relevance within a document is a potential indicator for the document-level relevance. Based on the relationship between passage-level and document-level relevance, we also show that utilizing passage-level relevance signals can improve existing document ranking models. This study helps us better understand how users perceive relevance for a document and inspire the designing of novel ranking models leveraging fine-grain, passage-level relevance signals.
Concepts,Types,and Uncertainty of "Relevance"in Information Retrieval

Wang Zhijin,Bian Dan,Wang Wenshuang

DOI: https://doi.org/10.3969/j.issn.1002-5197.2011.01.002

2011-01-01

Abstract:This paper reviews the four stages of relevance research,summarizes the various types of relevance,and classifies them as relevance of system-oriented,information-oriented and situation-oriented.Then,starts from the process of information retrieval,this paper analyses the uncertainty of various types of relevance in detail.Finally,predicts that the relevance research will pay attention to system,information and users at the same time.
When does Relevance Mean Usefulness and User Satisfaction in Web Search?

Jiaxin Mao,Yiqun Liu,Ke Zhou,Jian-Yun Nie,Jingtao Song,Min Zhang,Shaoping Ma,Jiashen Sun,Hengliang Luo

DOI: https://doi.org/10.1145/2911451.2911507

2016-01-01

Abstract:Relevance is a fundamental concept in information retrieval (IR) studies. It is however often observed that relevance as annotated by secondary assessors may not necessarily mean usefulness and satisfaction perceived by users. In this study, we confirm the difference by a laboratory study in which we collect relevance annotations by external assessors, usefulness and user satisfaction information by users, for a set of search tasks. We also find that a measure based on usefulness rather than relevance annotated has a better correlation with user satisfaction. However, we show that external assessors are capable of annotating usefulness when provided with more search context information. In addition, we also show that it is possible to generate automatically usefulness labels when some training data is available. Our findings explain why traditional system-centric evaluation metrics are not well aligned with user satisfaction and suggest that a usefulness-based evaluation method can be defined to better reflect the quality of search systems perceived by the users.
Explanation of Relevance Judgement Discrepancy with Quantum Interference.

Jun Wang,Dawei Song,Peng Zhang,Yuexian Hou,Peter Bruza

2010-01-01

Abstract:A key concept in many Information Retrieval (IR) tasks, e.g. document indexing, query language modelling, aspect and diversity retrieval, is the relevance measurement of topics, i.e. to what extent an information object (e.g. a document or a query) is about the topics. This paper investigates the interference of relevance measurement of a topic caused by another topic. For example, consider that two user groups are required to judge whether a topic q is relevant to a document d, and q is presented together with another topic (referred to as a companion topic). If different companion topics are used for different groups, interestingly different relevance probabilities of q given d can be reached. In this paper, we present empirical results showing that the relevance of a topic to a document is greatly affected by the companion topic’s relevance to the same document, and the extent of the impact differs with respect to different companion topics. We further analyse the phenomenon from classical and quantum-like interference perspectives, and connect the phenomenon to nonreality and contextuality in quantum mechanics. We demonstrate that quantum like model fits in the empirical data, could be potentially used for predicting the relevance when interference exists.
On Relevance in Information Retrieval Based on Relevance Theory

CHENG Ying,SUN Jian-jun,LI Bao-qiang

2010-01-01

Abstract:The relevance assessment module of information retrieval interaction model is annotated with information processing model of relevance theory.The conclusion of this paper is that it is feasible to annotate relevance assessment of information retrieval with information processing model of relevance theory.
Investigating Reading Behavior in Fine-grained Relevance Judgment

Zhijing Wu,Jiaxin Mao,Yiqun Liu,Min Zhang,Shaoping Ma

DOI: https://doi.org/10.1145/3397271.3401305

2020-01-01

Abstract:A better understanding of users' reading behavior helps improve many information retrieval (IR) tasks, such as relevance estimation and document ranking. Existing research has already leveraged eye movement information to investigate user's reading process during document-level relevance judgments and the findings were adopted to build more effective ranking models. Recently, fine-grained (e.g., passage or sentence level) relevance judgments have been paid much attention to with the requirements in conversational search and QA systems. However, there is still a lack of thorough investigation on user's reading behavior during these kinds of interaction processes. To shed light on this research question, we investigate how users allocate their attention to passages of a document during the relevance judgment process. With the eye-tracking data collected in a laboratory study, we show that users pay more attention to the "key" passages which contain key useful information. Users tend to revisit these key passages several times to accumulate and verify the gathered information. With both content and user behavior features, we find that key passages can be predicted with supervised learning. We believe that this work contributes to better understanding users' reading behavior and may provide more explainability for relevance estimation.
A Document Relevance Based Search Result Re-Ranking

周博,岑荣伟,刘奕群,张敏,金奕江,马少平

DOI: https://doi.org/10.3969/j.issn.1003-0077.2010.03.003

2010-01-01

Abstract:Relevance Feedback has been studied in information retrieval research for the past 30 years.It has been shown to be worthwhile in a wide variety of settings,either the actual user feedback is availableor it is implicit.Since the applications of relevance feedback and the type of user input to relevance feedback have changed in the Web environment,the relevance feedback is again emphasized by researchers.A document relevance based search result re-ranking approach is proposed in this paper,which makes use of both the relevant documents and irrelevant documents in feedback information.The approach is shown to be consistently valid for performance improvement on the standard large scale test dataset of TREC 2008 Relevance Feedback Track.
A Comparison of Retrieval Result Relevance Judgments Between American and Chinese Users

Jin Zhang,Yuehua Zhao,Xin Cai,Taowen Le,Wei Fei,Feicheng Ma

DOI: https://doi.org/10.4018/jgim.2020070108

2020-01-01

Journal of Global Information Management

Abstract:Relevance judgment plays an extremely significant role in information retrieval. This study investigates the differences between American users and Chinese users in relevance judgment during the information retrieval process. 384 sets of relevance scores with 50 scores in each set were collected from 16 American users and 16 Chinese users as they judged retrieval records from two major search engines based on 24 predefined search tasks from 4 domain categories. Statistical analyses reveal that there are significant differences between American assessors and Chinese assessors in relevance judgments. Significant gender differences also appear within both the American and the Chinese assessor groups. The study also revealed significant interactions among cultures, genders, and subject categories. These findings can enhance the understanding of cultural impact on information retrieval and can assist in the design of effective cross-language information retrieval systems.
Review on Relevance Criteria(2000-2010)

CHENG Ying

DOI: https://doi.org/10.3969/j.issn.1002-1965.2011.09.016

IF: 3.1759

2011-01-01

Journal of Intelligence

Abstract:Nearly 20 empirical studies to relevance criteria published between 2000 and 2010 are reviewed in this paper.The content of these studies mainly relates to relevance criteria of web information sources,multimedia,specific populations,and particular perspective and dynamic nature of relevance criteria.Results show that: the existing research has formed an effective relevance criteria research paradigm;researches are based on real information need;researches cover various topics;different contexts have similar relevance criteria;the dynamic nature of relevance criteria.
Understanding Reading Attention Distribution during Relevance Judgement.

Xiangsheng Li,Yiqun Liu,Jiaxin Mao,Zexue He,Min Zhang,Shaoping Ma

DOI: https://doi.org/10.1145/3269206.3271764

2018-01-01

Abstract:Reading is a complex cognitive activity in many information retrieval related scenarios, such as relevance judgement and question answering. There exists plenty of works which model these processes as a matching problem, which focuses on how to estimate the relevance score between a document and a query. However, little is known about what happened during the reading process, i.e., how users allocate their attention while reading a document during a specific information retrieval task. We believe that a better understanding of this process can help us design better weighting functions inside the document and contributes to the improvement of ranking performance. In this paper, we focus on the reading process during relevance judgement task. We designed a lab-based user study to investigate human reading patterns in assessing a document, where users' eye movements and their labeled relevant text were collected, respectively. Through a systematic analysis into the collected data, we propose a two-stage reading model which consists of a preliminary relevance judgement stage (Stage 1) and a reading with preliminary relevance stage (Stage 2). In addition, we investigate how different behavior biases affect users' reading behaviors in these two stages. Taking these biases into consideration, we further build prediction models for user's reading attention. Experiment results show that query independent features outperform query dependent features, which indicates that users allocate attentions based on many signals other than query terms in this process. Our study sheds light on the understanding of users' attention allocation during relevance judgement and provides implications for improving the design of existing ranking models.
A Linguistic Study on Relevance Modeling in Information Retrieval

Yixing Fan,Jiafeng Guo,Xinyu Ma,Ruqing Zhang,Yanyan Lan,Xueqi Cheng

DOI: https://doi.org/10.1145/3442381.3450009

2021-03-01

Abstract:Relevance plays a central role in information retrieval (IR), which has received extensive studies starting from the 20th century. The definition and the modeling of relevance has always been critical challenges in both information science and computer science research areas. Along with the debate and exploration on relevance, IR has already become a core task in many real-world applications, such as Web search engines, question answering systems, conversational bots, and so on. While relevance acts as a unified concept in all these retrieval tasks, the inherent definitions are quite different due to the heterogeneity of these tasks. This raises a question to us: Do these different forms of relevance really lead to different modeling focuses? To answer this question, in this work, we conduct an empirical study on relevance modeling in three representative IR tasks, i.e., document retrieval, answer retrieval, and response retrieval. Specifically, we attempt to study the following two questions: 1) Does relevance modeling in these tasks really show differences in terms of natural language understanding (NLU)? We employ 16 linguistic tasks to probe a unified retrieval model over these three retrieval tasks to answer this question. 2) If there do exist differences, how can we leverage the findings to enhance the relevance modeling? We proposed three intervention methods to investigate how to leverage different modeling focuses of relevance to improve these IR tasks. We believe the way we study the problem as well as our findings would be beneficial to the IR community.

Information Retrieval
On Relevance in Information Retrieval Based on Relevance Theory——information production,indexing

成颖,孙建军,李宝强

2010-01-01

Abstract:Based on the work of Saracevic and Harter,the authors argue that relevance theory can become the basic theory of relevance research in information retrieval.The two modules of information retrieval interaction model,information production and indexing,are annotated with relevance theory.
Searching Documents Based On Relevance And Type

Jun Xu,Yunbo Cao,Hang Li,Nick Craswell,Yalou Huang

DOI: https://doi.org/10.1007/978-3-540-71496-5_60

2007-01-01

Abstract:This paper extends previous work on document retrieval and document type classification, addressing the problem of 'typed search'. Specifically, given a query and a designated document type, the search system retrieves and ranks documents not only based on the relevance to the query, but also based on the likelihood of being the designated document type. The paper formalizes the problem in a general framework consisting of 'relevance model' and 'type model'. The relevance model indicates whether or not a document is relevant to a query. The type model indicates whether or not a document belongs to the designated document type. We consider three methods for combing the models: linear combination of scores, thresholding on the type score, and a hybrid of the previous two methods. We take course page search and instruction document search as examples and have conducted a series of experiments. Experimental results show our proposed approaches can significantly outperform the baseline methods.
Does Document Relevance Affect the Searcher's Perception of Time?

Cheng Luo,Yiqun Liu,Tetsuya Sakai,Ke Zhou,Fan Zhang,Xue Li,Shaoping Ma

DOI: https://doi.org/10.1145/3018661.3018694

2017-01-01

Abstract:Time plays an essential role in multiple areas of Information Retrieval (IR) studies such as search evaluation, user behavior analysis, temporal search result ranking and query understanding. Especially, in search evaluation studies, time is usually adopted as a measure to quantify users' efforts in search processes. Psychological studies have reported that the time perception of human beings can be affected by many stimuli, such as attention and motivation, which are closely related to many cognitive factors in search. Considering the fact that users' search experiences are affected by their subjective feelings of time, rather than the objective time measured by timing devices, it is necessary to look into the different factors that have impacts on search users' perception of time. In this work, we make a first step towards revealing the time perception mechanism of search users with the following contributions: (1) We establish an experimental research framework to measure the subjective perception of time while reading documents in search scenario, which originates from but is also different from traditional time perception measurements in psychological studies. (2) With the framework, we show that while users are reading result documents, document relevance has small yet visible effect on search users' perception of time. By further examining the impact of other factors, we demonstrate that the effect on relevant documents can also be influenced by individuals and tasks. (3) We conduct a preliminary experiment in which the difference between perceived time and dwell time is taken into consideration in a search evaluation task. We found that the revised framework achieved a better correlation with users' satisfaction feedbacks. This work may help us better understand the time perception mechanism of search users and provide insights in how to better incorporate time factor in search evaluation studies.
Order Effect in Relevance Judgment.

Yunjie Xu,Dong Wang

DOI: https://doi.org/10.1002/asi.20826

2008-01-01

Journal of the American Society for Information Science and Technology

Abstract:The order effect of relevance judgment refers to the different relevance perceptions of a document when it appears in different positions in a list. Although the order effect of relevance judgment has significant theoretical and practical implications, the extant literature is inconclusive regarding its existence and forming mechanisms. This study proposes a set of order effect forming mechanisms, including the learning effect, the subneed scheduling effect, and the cursoriness effect based on the conceptualization of dynamic relevance and the psychology of cognitive elaboration. Our empirical study indicates that in an interactive information retrieval setting, when a document list is reasonably long, order effects demonstrate a curvilinear pattern that conforms to the combined effect of the three mechanisms. Moreover, the curvilinear pattern of order effect could differ for documents of different relevance levels.
Novelty and Topicality in Interactive Information Retrieval

Yunjie Xu,Hainan Yin

DOI: https://doi.org/10.1002/asi.20709

2007-01-01

Journal of the American Society for Information Science and Technology

Abstract:The information science research community is characterized by a paradigm split, with a system-centered cluster working on information retrieval (IR) algorithms and a user-centered cluster working on user behavior. The two clusters rarely leverage each other's insight and strength. One major suggestion from user-centered studies is to treat the relevance judgment of documents as a subjective, multidimensional, and dynamic concept rather than treating it as objective and based on topicality only. This study explores the possibility to enhance users' topicality-based relevance judgment with subjective novelty judgment in interactive IR. A set of systems is developed which differs in the way the novelty judgment is incorporated. In particular, this study compares systems which assume that users' novelty judgment is directed to a certain subtopic area and those which assume that users' novelty judgment is undirected. This study also compares systems which assume that users judge a document based on topicality first and then novelty in a stepwise, noncompensatory fashion and those which assume that users consider topicality and novelty simultaneously and as compensatory to each other. The user study shows that systems assuming directed novelty in general have higher relevance precision, but systems assuming a stepwise judgment process and systems assuming a compensatory judgment process are not significantly different. © 2008 Wiley Periodicals, Inc.

Relevance Judgment: What Do Information Users Consider Beyond Topicality?

Relevance Judgment in Epistemic and Hedonic Information Searches

User-Oriented Relevance Judgment: A Conceptual Model

Understanding Relevance Judgments in Legal Case Retrieval

Relevance judgments of mobile commercial information

Investigating Passage-level Relevance and Its Role in Document-level Relevance Judgment

Concepts,Types,and Uncertainty of "Relevance"in Information Retrieval

When does Relevance Mean Usefulness and User Satisfaction in Web Search?

Explanation of Relevance Judgement Discrepancy with Quantum Interference.

On Relevance in Information Retrieval Based on Relevance Theory

Investigating Reading Behavior in Fine-grained Relevance Judgment

A Document Relevance Based Search Result Re-Ranking

A Comparison of Retrieval Result Relevance Judgments Between American and Chinese Users

Review on Relevance Criteria(2000-2010)

Understanding Reading Attention Distribution during Relevance Judgement.

A Linguistic Study on Relevance Modeling in Information Retrieval

On Relevance in Information Retrieval Based on Relevance Theory——information production,indexing

Searching Documents Based On Relevance And Type

Does Document Relevance Affect the Searcher's Perception of Time?

Order Effect in Relevance Judgment.

Novelty and Topicality in Interactive Information Retrieval