Abstract:Mobile search engine result pages (SERPs) are becoming highly visual and heterogenous. Unlike the traditional ten-blue-link SERPs for desktop search, different verticals and cards occupy different amounts of space within the small screen. Hence, traditional retrieval measures that regard the SERP as a ranked list of homogeneous items are not adequate for evaluating the overall quality of mobile SERPs. Specifically, we address the following new problems in mobile search evaluation: (1) Different retrieved items have different heights within the scrollable SERP, unlike a ten-blue-link SERP in which results have similar heights with each other. Therefore, the traditional rank-based decaying functions are not adequate for mobile search metrics. (2) For some types of verticals and cards, the information that the user seeks is already embedded in the snippet, which makes clicking on those items to access the landing page unnecessary. (3) For some results with complex sub-components (and usually a large height), the total gain of the results cannot be obtained if users only read part of their contents. The benefit brought by the result is affected by user's reading behavior and the internal gain distribution (over the height) should be modeled to get a more accurate estimation. To tackle these problems, we conduct a lab-based user study to construct suitable user behavior model for mobile search evaluation. From the results, we find that the geometric heights of user's browsing trails can be adopted as a good signal of user effort. Based on these findings, we propose a new evaluation metric, Height-Biased Gain, which is calculated by summing up the product of gain distribution and discount factors that are both modeled in terms of result height. To evaluate the effectiveness of the proposed metric, we compare the agreement of evaluation metrics with side-by-side user preferences on a test collection composed of four mobile search engines. Experimental results show that HBG agrees with user preferences 85.33% of the time, which is better than all existing metrics.

Investigating Fine-Grained Usefulness Perception Process in Mobile Search.

Investigating Result Usefulness in Mobile Search.

Investigating Examination Behavior in Mobile Search

A Large-Scale Study of Mobile Search Examination Behavior.

A Two-Stage Model for User's Examination Behavior in Mobile Search.

Comparison of Perceptual Differences Between Users and Designers in Mobile Shopping App Interface Design: Implications for Evaluation Practice

Constructing Click Models for Mobile Search

User Preference Prediction in Mobile Search.

Constructing Click Model for Mobile Search with Viewport Time.

Efficient browsing of Web search results on mobile devices based on block importance model

Understanding and Predicting Usefulness Judgment in Web Search

Why Don't You Click: Understanding Non-Click Results inWeb Search with Brain Signals

Evaluating Mobile Search with Height-Biased Gain

Construction of information search behavior based on data mining

Why Don't You Click: Neural Correlates of Non-Click Behaviors in Web Search

Investigating Smartphone User Differences in Their Application Usage Behaviors: an Empirical Study

Using Information Scent to Understand Mobile and Desktop Web Search Behavior

An F-shape Click Model for Information Retrieval on Multi-block Mobile Pages

Behavior Modeling for Point of Interest Search

Empirical Research on User Acceptance of Mobile Searches

Between Clicks and Satisfaction: Study on Multi-Phase User Preferences and Satisfaction for Online News Reading.