On Relevance in Information Retrieval Based on Relevance Theory——information production,indexing

成颖,孙建军,李宝强

2010-01-01

Abstract:Based on the work of Saracevic and Harter,the authors argue that relevance theory can become the basic theory of relevance research in information retrieval.The two modules of information retrieval interaction model,information production and indexing,are annotated with relevance theory.

What problem does this paper attempt to address?

On Relevance in Information Retrieval Based on Relevance Theory

CHENG Ying,SUN Jian-jun,LI Bao-qiang

2010-01-01

Abstract:The relevance assessment module of information retrieval interaction model is annotated with information processing model of relevance theory.The conclusion of this paper is that it is feasible to annotate relevance assessment of information retrieval with information processing model of relevance theory.
A Linguistic Study on Relevance Modeling in Information Retrieval

Yixing Fan,Jiafeng Guo,Xinyu Ma,Ruqing Zhang,Yanyan Lan,Xueqi Cheng

DOI: https://doi.org/10.1145/3442381.3450009

2021-03-01

Abstract:Relevance plays a central role in information retrieval (IR), which has received extensive studies starting from the 20th century. The definition and the modeling of relevance has always been critical challenges in both information science and computer science research areas. Along with the debate and exploration on relevance, IR has already become a core task in many real-world applications, such as Web search engines, question answering systems, conversational bots, and so on. While relevance acts as a unified concept in all these retrieval tasks, the inherent definitions are quite different due to the heterogeneity of these tasks. This raises a question to us: Do these different forms of relevance really lead to different modeling focuses? To answer this question, in this work, we conduct an empirical study on relevance modeling in three representative IR tasks, i.e., document retrieval, answer retrieval, and response retrieval. Specifically, we attempt to study the following two questions: 1) Does relevance modeling in these tasks really show differences in terms of natural language understanding (NLU)? We employ 16 linguistic tasks to probe a unified retrieval model over these three retrieval tasks to answer this question. 2) If there do exist differences, how can we leverage the findings to enhance the relevance modeling? We proposed three intervention methods to investigate how to leverage different modeling focuses of relevance to improve these IR tasks. We believe the way we study the problem as well as our findings would be beneficial to the IR community.

Information Retrieval
Relevance Judgment: What Do Information Users Consider Beyond Topicality?

YJ Xu,ZW Chen

DOI: https://doi.org/10.1002/asi.20361

2006-01-01

Journal of the American Society for Information Science and Technology

Abstract:How does an information user perceive a document as relevant? The literature on relevance has identified numerous factors affecting such a judgment. Taking a cognitive approach, this study focuses ...
Relevance Judgment in Epistemic and Hedonic Information Searches

Yunjie Xu

DOI: https://doi.org/10.1002/asi.20461

2006-01-01

Journal of the American Society for Information Science and Technology

Abstract:Research in information science now regards users' relevance judgment as subjective perception. However, user-centered studies in the extant literature mainly focus on relevance judgment in problem solving contexts in which the situational relevance of a document is the main concern for users. This study investigates users' relevance judgment in non-problem-solving contexts, i.e., when users search information for epistemic value or entertainment. It is posited that informative relevance and affective relevance should be the main concerns for users. Based on H. P. Grice's (1975, 1989) communication theory and Y. Xu and Z. Chen's (2006) framework, this study tests the significance of topicality, novelty, reliability, understandability, and scope to informative relevance and affective relevance in non-problem-solving contexts. This empirical study finds novelty, reliability, and topicality to be key aspects of informative relevance. © 2007 Wiley Periodicals, Inc.
When does Relevance Mean Usefulness and User Satisfaction in Web Search?

Jiaxin Mao,Yiqun Liu,Ke Zhou,Jian-Yun Nie,Jingtao Song,Min Zhang,Shaoping Ma,Jiashen Sun,Hengliang Luo

DOI: https://doi.org/10.1145/2911451.2911507

2016-01-01

Abstract:Relevance is a fundamental concept in information retrieval (IR) studies. It is however often observed that relevance as annotated by secondary assessors may not necessarily mean usefulness and satisfaction perceived by users. In this study, we confirm the difference by a laboratory study in which we collect relevance annotations by external assessors, usefulness and user satisfaction information by users, for a set of search tasks. We also find that a measure based on usefulness rather than relevance annotated has a better correlation with user satisfaction. However, we show that external assessors are capable of annotating usefulness when provided with more search context information. In addition, we also show that it is possible to generate automatically usefulness labels when some training data is available. Our findings explain why traditional system-centric evaluation metrics are not well aligned with user satisfaction and suggest that a usefulness-based evaluation method can be defined to better reflect the quality of search systems perceived by the users.
User-Oriented Relevance Judgment: A Conceptual Model

Zhiwei Chen,Yunjie Xu

DOI: https://doi.org/10.1109/HICSS.2005.654

2005-01-01

Abstract:The concept of relevance has been heatedly debated in last decade. Not satisfied with the narrow and technical definition of system relevance, researchers turn to the subjective and situational aspect of this concept. How does a user perceive a document as relevant? The literature on relevance has identified numerous factors affecting such judgment. Taking a cognitive approach, this study focuses on the criteria users employ in making relevance judgment. Based on Grice's theory of communication, this paper proposes a five-factor model of relevance: topicality, novelty, reliability, understandability, and scope. Data are collected from a semi-controlled survey study and analyzed following a psychometric procedure. The result supports topicality and novelty as the key relevance criteria. Theoretical and practical implications of this study are discussed.
On Models of Information Retrieval Processes

CT YU,WS LUK,MK SIU

DOI: https://doi.org/10.1016/0306-4379(79)90004-8

IF: 3.18

1979-01-01

Information Systems

Abstract:A model called the binary independence model is presented as a generalization of a few models which have applied to the analyses of a clustered search process, some indexing strategies and a relevance feedback process. This model, together with the Swets model, the linked-2-Poisson model, the 2-Poisson model, the binary limited-dependence model, the tree dependence model, the binary dependence model and the non-binary independent model are compared and contrasted. Despite the fact that these models are intended for different applications, three aspects of modelling are identified, based on which, these models are compared. The three aspects are the class concept, the distribution of similarities and the relation between a matching function and its retrieval effectiveness. As results of the comparison between the models, more insight into the models is gained and a set of guidelines are suggested to help a system designer to choose a model.
A Relevance Evaluation Model of Information Retrieval Based on Weighted Term Frequency

ZHANG Chun-qing,CHEN Chao,SHAO Zheng-rong,YU Neng-hai

DOI: https://doi.org/10.3969/j.issn.1006-9348.2008.01.035

2008-01-01

Abstract:Relevance evaluation model is an important research issue in the field of information retrieval.The bas- ic information retrieval models are boolean model,vector space model and probabilistic model.The latter two models are implemented in many retrieval systems extensively but the different position of query term in every document is ig- nored.Some researches have considered the information HTML tags but the scheme of assigning weighted parameters is not ideal.In this paper,WTFM(Weighted Term Frequency Model)is proposed according to the existence of term frequency(TF).And these weighted coefficients are learned by simulated annealing algorithm.The results of the ex- periments show that the introduction of TF's weights brings improvements to the system.
On relevance theory and translation of literary works

Zhang Lu-yan

Abstract:Relevance theory is the basic theory of cognitive pragmatics.It has a great influence on translation including translation of literary works since Gutt first introduced it to translation field.However,relevance theory has met challenges when facing the speciality of literary works which are rich of cultural connotations.Therefore,their relation is very complicated because the explanation of relevance theory on translation of literary works can be accepted but not very convincing.

Linguistics,Computer Science
Review on Relevance Criteria(2000-2010)

CHENG Ying

DOI: https://doi.org/10.3969/j.issn.1002-1965.2011.09.016

IF: 3.1759

2011-01-01

Journal of Intelligence

Abstract:Nearly 20 empirical studies to relevance criteria published between 2000 and 2010 are reviewed in this paper.The content of these studies mainly relates to relevance criteria of web information sources,multimedia,specific populations,and particular perspective and dynamic nature of relevance criteria.Results show that: the existing research has formed an effective relevance criteria research paradigm;researches are based on real information need;researches cover various topics;different contexts have similar relevance criteria;the dynamic nature of relevance criteria.
Understanding Relevance Judgments in Legal Case Retrieval

Yunqiu Shao,Yueyue Wu,Yiqun Liu,Jiaxin Mao,Shaoping Ma

DOI: https://doi.org/10.1145/3569929

IF: 4.657

2023-02-07

ACM Transactions on Information Systems

Abstract:Legal case retrieval, which aims to retrieve relevant cases given a query case, has drawn increasing research attention in recent years. While much research has worked on developing automatic retrieval models, how to characterize relevance in this specialized information retrieval (IR) task is still an open question. Towards an in-depth understanding of relevance judgments, we conduct a laboratory user study that involves 72 participants of different domain expertise. In the user study, we collect the relevance score along with detailed explanations for the relevance judgment and various measures of the judgment process. From the collected data, we observe that both the subjective (e.g., domain expertise) and objective (e.g., query/case property) factors influence the relevance judgment process. By investigating the collected user explanations, we identify task-specific patterns of user attention distribution and re-think the criteria for relevance judgments. Moreover, we investigate the similarity in attention distribution between models and users. Further, we propose a two-stage framework that utilizes user attention to improve relevance estimation for legal case retrieval. Our study sheds light on understanding relevance judgments in legal case retrieval and provides implications for improving the design of corresponding retrieval systems.

computer science, information systems
Relevance Estimation with Multiple Information Sources on Search Engine Result Pages.

Junqi Zhang,Yiqun Liu,Shaoping Ma,Qi Tian

DOI: https://doi.org/10.1145/3269206.3271673

2018-01-01

Abstract:Relevance estimation is among the most important tasks in the ranking of search results because most search engines follow the Probability Ranking Principle. Current relevance estimation methodologies mainly concentrate on text matching between the query and Web documents, link analysis and user behavior models. However, users judge the relevance of search results directly from Search Engine Result Pages (SERPs), which provide valuable signals for reranking. Morden search engines aggregate heterogeneous information items (such as images, news, and hyperlinks) to a single ranking list on SERPs. The aggregated search results have different visual patterns, textual semantics and presentation structures, and a better strategy should rely on all these information sources to improve ranking performance. In this paper, we propose a novel framework named Joint Relevance Estimation model (JRE), which learns the visual patterns from screenshots of search results, explores the presentation structures from HTML source codes and also adopts the semantic information of textual contents. To evaluate the performance of the proposed model, we construct a large scale practical Search Result Relevance (SRR) dataset which consists of multiple information sources and 4-grade relevance scores of over 60,000 search results. Experimental results show that the proposed JRE model achieves better performance than state-of-the-art ranking solutions as well as the original ranking of commercial search engines.
Relevance Feedback Technologies in Search Engines

陶跃华,孙茂松

DOI: https://doi.org/10.3969/j.issn.1000-7490.2001.04.020

2001-01-01

Abstract:As an important component of search engines, the relevance feedback system is very effective for improving the performance of search engines. This paper firstly reviews the history of relevance feedback technology in the past 30 years, then introduces 2 major methods in relevance feedback, i. e. term reweighting and query expansion, and discusses the relevance feedback technologies based on vector space model and statistical ranking model.
A Document Relevance Based Search Result Re-Ranking

周博,岑荣伟,刘奕群,张敏,金奕江,马少平

DOI: https://doi.org/10.3969/j.issn.1003-0077.2010.03.003

2010-01-01

Abstract:Relevance Feedback has been studied in information retrieval research for the past 30 years.It has been shown to be worthwhile in a wide variety of settings,either the actual user feedback is availableor it is implicit.Since the applications of relevance feedback and the type of user input to relevance feedback have changed in the Web environment,the relevance feedback is again emphasized by researchers.A document relevance based search result re-ranking approach is proposed in this paper,which makes use of both the relevant documents and irrelevant documents in feedback information.The approach is shown to be consistently valid for performance improvement on the standard large scale test dataset of TREC 2008 Relevance Feedback Track.
An Explanation for the Relational Semantics of Relevance Logic

Zhou Beihai,Jia Qing

DOI: https://doi.org/10.3969/j.issn.1674-3202.2015.01.005

2015-01-01

Abstract:Relational semantics is one of the most popular semantics for relevance logic. How-ever, because this semantics is a kind of“purely”formal semantics, the ternary relation R lacks intuition, and that is why there are various interpretations for relational seman-tics. As to our opinion, R is a representation of the relation among rules, premises and conclusions. Based on this opinion, inference semantics is presented. The background of inference semantics is the formal structure of inference and this feature makes inference semantics intuitive for our understanding. In this paper, we will prove that relational semantics and inference semantics are equivalent with respect to the relevance logical system R+. From this equivalence relation, inference semantics could be viewed as an intermediate link for the intuitive interpretation of relational semantics. By inference semantics, relational semantics could got an interpretation from the structure of infer-ence. Relevance logic is a logic of inference, so inference semantics is coincident with the history of relevance logic.
A Theory of Information Matching

Jagadeesh Gorla,Stephen Robertson,Jun Wang,Tamas Jambor

DOI: https://doi.org/10.48550/arXiv.1205.5569

2012-06-02

Abstract:In this work, we propose a theory for information matching. It is motivated by the observation that retrieval is about the relevance matching between two sets of properties (features), namely, the information need representation and information item representation. However, many probabilistic retrieval models rely on fixing one representation and optimizing the other (e.g. fixing the single information need and tuning the document) but not both. Therefore, it is difficult to use the available related information on both the document and the query at the same time in calculating the probability of relevance. In this paper, we address the problem by hypothesizing the relevance as a logical relationship between the two sets of properties; the relationship is defined on two separate mappings between these properties. By using the hypothesis we develop a unified probabilistic relevance model which is capable of using all the available information. We validate the proposed theory by formulating and developing probabilistic relevance ranking functions for both ad-hoc text retrieval and collaborative filtering. Our derivation in text retrieval illustrates the use of the theory in the situation where no relevance information is available. In collaborative filtering, we show that the resulting recommender model unifies the user and item information into a relevance ranking function without applying any dimensionality reduction techniques or computing explicit similarity between two different users (or items), in contrast to the state-of-the-art recommender models.

Information Retrieval
Research on Knowledge Representation and Knowledge Relevance Measurement

Yang Jianlin

DOI: https://doi.org/10.16353/j.cnki.1000-7490.2011.05.015

2011-01-01

Abstract:Mining and organizing knowledge element is the basis of knowledge indexing,but it's a very difficult work.This paper summarizes the relevant research results of knowledge indexing from the perspective of knowledge element representation,knowledge retrieval and knowledge relevance measurement,and propose some own ideas.
Modelling Dynamic Interactions between Relevance Dimensions

Sagar Uprety,Shahram Dehdashti,Lauren Fell,Peter Bruza,Dawei Song

DOI: https://doi.org/10.48550/arXiv.1907.10943

2019-07-25

Abstract:Relevance is an underlying concept in the field of Information Science and Retrieval. It is a cognitive notion consisting of several different criteria or dimensions. Theoretical models of relevance allude to interdependence between these dimensions, where their interaction and fusion leads to the final inference of relevance. We study the interaction between the relevance dimensions using the mathematical framework of Quantum Theory. It is considered a generalised framework to model decision making under uncertainty, involving multiple perspectives and influenced by context. Specifically, we conduct a user study by constructing the cognitive analogue of a famous experiment in Quantum Physics. The data is used to construct a complex-valued vector space model of the user's cognitive state, which is used to explain incompatibility and interference between relevance dimensions. The implications of our findings to inform the design of Information Retrieval systems are also discussed.

Information Retrieval
Usefulness as the Criterion for Evaluation of Interactive Information Retrieval

M. Cole,J. Liu,N. J. Belkin,R. Bierig,J. Gwizdka,C. Liu,J. Zhang,X. Zhang

2009-01-01

Abstract:The purpose of an information retrieval (IR) system is to help users accomplish a task. IR system evaluation should consider both task success and the value of support given over the entire information seeking episode. Relevance-based measurements fail to address these requirements. In this paper, usefulness is proposed as a basis for IR evaluation.
Investigating Bell Inequalities for Multidimensional Relevance Judgments in Information Retrieval

Sagar Uprety,Dimitris Gkoumas,Dawei Song

DOI: https://doi.org/10.48550/arXiv.1811.06645

2019-03-14

Abstract:Relevance judgment in Information Retrieval is influenced by multiple factors. These include not only the topicality of the documents but also other user oriented factors like trust, user interest, etc. Recent works have identified these various factors into seven dimensions of relevance. In a previous work, these relevance dimensions were quantified and user's cognitive state with respect to a document was represented as a state vector in a Hilbert Space, with each relevance dimension representing a basis. It was observed that relevance dimensions are incompatible in some documents, when making a judgment. Incompatibility being a fundamental feature of Quantum Theory, this motivated us to test the Quantum nature of relevance judgments using Bell type inequalities. However, none of the Bell-type inequalities tested have shown any violation. We discuss our methodology to construct incompatible basis for documents from real world query log data, the experiments to test Bell inequalities on this dataset and possible reasons for the lack of violation.

Information Retrieval

On Relevance in Information Retrieval Based on Relevance Theory——information production,indexing

On Relevance in Information Retrieval Based on Relevance Theory

A Linguistic Study on Relevance Modeling in Information Retrieval

Relevance Judgment: What Do Information Users Consider Beyond Topicality?

Relevance Judgment in Epistemic and Hedonic Information Searches

When does Relevance Mean Usefulness and User Satisfaction in Web Search?

User-Oriented Relevance Judgment: A Conceptual Model

On Models of Information Retrieval Processes

A Relevance Evaluation Model of Information Retrieval Based on Weighted Term Frequency

On relevance theory and translation of literary works

Review on Relevance Criteria(2000-2010)

Understanding Relevance Judgments in Legal Case Retrieval

Relevance Estimation with Multiple Information Sources on Search Engine Result Pages.

Relevance Feedback Technologies in Search Engines

A Document Relevance Based Search Result Re-Ranking

An Explanation for the Relational Semantics of Relevance Logic

A Theory of Information Matching

Research on Knowledge Representation and Knowledge Relevance Measurement

Modelling Dynamic Interactions between Relevance Dimensions

Usefulness as the Criterion for Evaluation of Interactive Information Retrieval

Investigating Bell Inequalities for Multidimensional Relevance Judgments in Information Retrieval