Abstract:The effectiveness of an IR system is gauged not just by its ability to retrieve relevant results but also by how it presents these results to users; an engaging presentation often correlates with increased user satisfaction. While existing research has delved into the link between user satisfaction, IR performance metrics, and presentation, these aspects have typically been investigated in isolation. Our research aims to bridge this gap by examining the relationship between query performance, presentation and user satisfaction. For our analysis, we conducted a between-subjects experiment comparing the effectiveness of various result card layouts for an ad-hoc news search interface. Drawing data from the TREC WaPo 2018 collection, we centered our study on four specific topics. Within each of these topics, we assessed six distinct queries with varying nDCG values. Our study involved 164 participants who were exposed to one of five distinct layouts containing result cards, such as "title'', "title+image'', or "title+image+summary''. Our findings indicate that while nDCG is a strong predictor of user satisfaction at the query level, there exists no linear relationship between the performance of the query, presentation of results and user satisfaction. However, when considering the total gain on the initial result page, we observed that presentation does play a significant role in user satisfaction (at the query level) for certain layouts with result cards such as, title+image or title+image+summary. Our results also suggest that the layout differences have complex and multifaceted impacts on satisfaction. We demonstrate the capacity to equalize user satisfaction levels between queries of varying performance by changing how results are presented. This emphasizes the necessity to harmonize both performance and presentation in IR systems, considering users' diverse preferences.
What problem does this paper attempt to address?
This paper attempts to address the issue of how query performance and presentation in information retrieval (IR) systems jointly affect user satisfaction. Specifically, the researchers focus on the impact of query performance metrics (such as nDCG) and presentation methods (such as combinations of titles, images, summaries, etc.) on user satisfaction under different result card layouts.
### Main Research Questions:
1. **How do the quality of query results (measured by query performance) and interface layout affect user satisfaction in information retrieval tasks?**
2. **What is the impact of different interface layouts on user satisfaction, including overall satisfaction, engine preference, productivity, and mental effort?**
### Research Background:
- **System-side Costs**: Includes standard IR effectiveness metrics such as precision, cumulative gain (CG), discounted cumulative gain (DCG), and normalized discounted cumulative gain (nDCG).
- **User-side Costs**: Includes query formulation, result display, browsing time, user satisfaction, etc.
- **Existing Research**: Most studies either independently examine the impact of query performance on user satisfaction or independently examine the impact of result presentation on user satisfaction, but few studies consider the combined impact of both aspects.
### Methodology:
- **Experimental Design**: A between-subjects design was used, with participants randomly assigned to 5 different interface layouts, each containing different types of result cards (e.g., title+image+summary, title only, etc.).
- **Data Source**: News articles from the TREC WaPo 2018 dataset were used, with four specific topics selected for the study.
- **Participants**: A total of 164 participants, primarily aged between 20 and 40, including both students and non-students.
- **Task**: Participants were required to complete news search tasks based on given topics and mark relevant documents.
### Key Findings:
- **Query Performance (nDCG)**: At the query level, nDCG is a strong predictor that significantly affects user satisfaction.
- **Result Presentation**: For certain layouts (e.g., title+image or title+image+summary), the total gain on the initial result page significantly impacts user satisfaction.
- **Complexity**: The impact of different layouts on user satisfaction is multifaceted and complex. By changing the result presentation, it is possible to balance user satisfaction across different query performances to some extent.
### Conclusion:
- **Integrated Performance and Presentation**: To improve user satisfaction, IR systems need to find a balance between query performance and result presentation, considering the diverse preferences of users.
- **Future Directions**: The findings can guide the optimization of future IR systems, emphasizing the balance between system performance and result presentation.
In summary, this paper reveals the combined impact of query performance and result presentation on user satisfaction through empirical research, providing valuable insights for designing more user-friendly IR systems.