Decoy Effect In Search Interaction: Understanding User Behavior and Measuring System Vulnerability

Nuo Chen,Jiqun Liu,Hanpei Fang,Yuankai Luo,Tetsuya Sakai,Xiao-Ming Wu

2024-10-08

Abstract:This study examines the decoy effect's underexplored influence on user search interactions and methods for measuring information retrieval (IR) systems' vulnerability to this effect. It explores how decoy results alter users' interactions on search engine result pages, focusing on metrics like click-through likelihood, browsing time, and perceived document usefulness. By analyzing user interaction logs from multiple datasets, the study demonstrates that decoy results significantly affect users' behavior and perceptions. Furthermore, it investigates how different levels of task difficulty and user knowledge modify the decoy effect's impact, finding that easier tasks and lower knowledge levels lead to higher engagement with target documents. In terms of IR system evaluation, the study introduces the DEJA-VU metric to assess systems' susceptibility to the decoy effect, testing it on specific retrieval tasks. The results show differences in systems' effectiveness and vulnerability, contributing to our understanding of cognitive biases in search behavior and suggesting pathways for creating more balanced and bias-aware IR evaluations.

Information Retrieval

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This study aims to explore the following two aspects:

1. **The Impact of the Decoy Effect on User Search Interaction**:
- The decoy effect is a cognitive bias where the introduction of an inferior option changes the user's preference between two initial options. The study focuses on how this effect influences user search behavior, particularly in terms of click-through rate, browsing dwell time, and perceived usefulness of documents.

2. **Measuring the Vulnerability of Information Retrieval Systems**:
- The study also explores how to assess the vulnerability of information retrieval systems to the decoy effect. Specifically, it proposes a new evaluation metric (DEcoy Judgement and Assessment VUlnerability, abbreviated as DEJA-VU) to measure whether retrieval models are prone to triggering the decoy bias.

### Research Background and Motivation

- **Cognitive Biases and User Behavior**: Traditional information retrieval models assume users are rational decision-makers, but in reality, user behavior is often influenced by various cognitive biases. The decoy effect is one such important cognitive bias, which has been widely studied in shopping decisions but has not been fully explored in the field of information retrieval.

- **Limitations of Existing Research**: Although Eickhoff et al. studied the impact of decoy documents on crowd-sourced assessors, these studies mainly focused on assessors within annotation interfaces and did not involve user interaction on search engine result pages (SERP).

### Research Methods and Experimental Design

- **Datasets and Experimental Design**: The study is based on three publicly available user behavior datasets. Through regression analysis methods, it analyzes the impact of decoy documents on user click-through rate, browsing time, and perceived usefulness.

- **Task Difficulty and User Knowledge Level**: The study further explores the differences in the impact of the decoy effect under different task difficulties and user knowledge levels.

### Main Contributions

1. **Extending the Understanding of the Decoy Effect**: This is the first study to apply the decoy effect to the field of interactive information retrieval, providing empirical evidence on how decoy documents influence user click patterns, browsing time, and perceived usefulness of documents.

2. **User Behavior in Different Contexts**: The study reveals the different impacts of the decoy effect on user behavior under varying task difficulties and user knowledge levels, enriching the research on how tasks and search tasks influence user information seeking and search behavior.

3. **System Vulnerability Evaluation Metric**: It proposes a new evaluation metric (DEJA-VU) to assess the vulnerability of text retrieval systems to the decoy effect, helping developers improve existing systems by enhancing effectiveness while reducing sensitivity to the decoy effect.

### Conclusion

Through experimental results, the study finds that:

- The presence of decoy documents significantly increases the likelihood of documents being clicked and their perceived usefulness.
- When task difficulty is higher, users are less likely to click on documents with decoys, spend less time, and give lower ratings.
- Users with lower knowledge levels about the search topic are more likely to give higher usefulness ratings to documents with decoys.
- DEJA-VU scores indicate that models like ColBERT and SPLADE++ maintain high relevance while being less sensitive to the decoy effect.

### Summary

This study not only extends the understanding of the decoy effect in the field of interactive information retrieval but also provides new tools and methods for evaluating and improving the vulnerability of information retrieval systems.