Abstract:User satisfaction is an important variable in Web search evaluation studies and has received more and more attention in recent years. Many studies regard user satisfaction as the ground truth for designing better evaluation metrics. However, most of the existing studies focus on designing Cranfield-like evaluation metrics to reflect user satisfaction at query-level. As information need becomes more and more complex, users often need multiple queries and multi-round search interactions to complete a search task (e.g. exploratory search). In those cases, how to characterize the user's satisfaction during a search session still remains to be investigated. In this paper, we collect a dataset through a laboratory study in which users need to complete some complex search tasks. With the help of hierarchical linear models (HLM), we try to reveal how user's query-level and session-level satisfaction are affected by different cognitive effects. A number of interesting findings are made. At query level, we found that although the relevance of top-ranked documents have important impacts (primacy effect), the average/maximum of perceived usefulness of clicked documents is a much better sign of user satisfaction. At session level, perceived satisfaction for a particular query is also affected by the other queries in the same session (anchor effect or expectation effect). We also found that session-level satisfaction correlates mostly with the last query in the session (recency effect). The findings will help us design better session-level user behavior models and corresponding evaluation metrics.

Geometric session detection method for sogou log

Query Subtopic Mining Via Subtractive Initialization of Non-negative Sparse Latent Semantic Analysis

Sogou Query Log Analysis: A Case Study for Collaborative Recommendation or Personalized IR

The Characteristic of Chinese Search Based on Sogou Log

Research on System Log Anomaly Detection Combining Two-way Slice GRU and GA-Attention Mechanism

A Novel Quality Evaluating Method for Over-Segmentation Approaches Using Real-Time Boundary Information

Research on User Behavior Based on Session Analysis

Investigating Session Search Behavior with Knowledge Graphs

Utilizing Variability of Time and Term Content, Within and Across Users in Session Detection.

Swarm Intelligence Based Topic Identification for Sessions in Web Access Log

Towards Designing Better Session Search Evaluation Metrics

SogouQ: The First Large-Scale Test Collection with Click Streams Used in a Shared-Task Evaluation

Investigating the Role of Eye Movements and Physiological Signals in Search Satisfaction Prediction Using Geometric Analysis

Revealing Secrets in SPARQL Session Level

Investigating Cognitive Effects in Session-level Search User Satisfaction

Sogou-ST: A New Dataset with Large-scale Refined Real-world Web Search Sessions

Computing Geographical Serving Area Based On Search Logs And Website Categorization

User Behaviors Lend a Helping Hand: Learning Paraphrase Query Patterns from Search Log Sessions.

A New Algorithm for Inferring User Search Goals with Feedback Sessions

Discovering Related Web Queries Based on Search Engine's User Log

Detecting Geographical Serving Area of Web Resources.