See or Hear? Exploring the Effect of Visual and Audio Hints and Gaze-assisted Task Feedback for Visual Search Tasks in Augmented Reality

Yuchong Zhang,Adam Nowak,Yueming Xuan,Andrzej Romanowski,Morten Fjeld

DOI: https://doi.org/10.1109/ISMAR59233.2023.00128

2023-11-14

Abstract:Augmented reality (AR) is emerging in visual search tasks for increasingly immersive interactions with virtual objects. We propose an AR approach providing visual and audio hints along with gaze-assisted instant post-task feedback for search tasks based on mobile head-mounted display (HMD). The target case was a book-searching task, in which we aimed to explore the effect of the hints together with the task feedback with two hypotheses. H1: Since visual and audio hints can positively affect AR search tasks, the combination outperforms the individuals. H2: The gaze-assisted instant post-task feedback can positively affect AR search tasks. The proof-of-concept was demonstrated by an AR app in HMD and a comprehensive user study (n=96) consisting of two sub-studies, Study I (n=48) without task feedback and Study II (n=48) with task feedback. Following quantitative and qualitative analysis, our results partially verified H1 and completely verified H2, enabling us to conclude that the synthesis of visual and audio hints conditionally improves the AR visual search task efficiency when coupled with task feedback.

Human-Computer Interaction

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve

This paper aims to explore the effects of visual cues, audio cues, and eye-tracking-assisted real-time task feedback on visual search tasks in augmented reality (AR) environments. Specifically, the study focuses on the following two hypotheses:

1. **H1**: The combination of visual and audio cues has a positive impact on AR visual search tasks, and this combined effect is superior to using either type of cue alone.
2. **H2**: Eye-tracking-assisted real-time task feedback has a positive effect on the efficiency of AR visual search tasks and reduces perceived workload.

To test these two hypotheses, the researchers designed an AR application based on a mobile head-mounted display (HMD) and conducted a comprehensive user study, which included two sub-studies:
- **Study I**: Experiments conducted without task feedback;
- **Study II**: Experiments conducted with task feedback.

Through quantitative and qualitative analysis methods, the study partially validated H1 and fully validated H2. The final conclusion indicates that incorporating task feedback, in addition to combining visual and audio cues, can significantly improve the efficiency of AR visual search tasks.