Multisensory extended reality applications offer benefits for volumetric biomedical image analysis in research and medicine

Kathrin Krieger,Jan Egger,Jens Kleesiek,Matthias Gunzer,Jianxu Chen

DOI: https://doi.org/10.1007/s10278-024-01094-x

2024-06-14

Abstract:3D data from high-resolution volumetric imaging is a central resource for diagnosis and treatment in modern medicine. While the fast development of AI enhances imaging and analysis, commonly used visualization methods lag far behind. Recent research used extended reality (XR) for perceiving 3D images with visual depth perception and touch but used restrictive haptic devices. While unrestricted touch benefits volumetric data examination, implementing natural haptic interaction with XR is challenging. The research question is whether a multisensory XR application with intuitive haptic interaction adds value and should be pursued. In a study, 24 experts for biomedical images in research and medicine explored 3D medical shapes with 3 applications: a multisensory virtual reality (VR) prototype using haptic gloves, a simple VR prototype using controllers, and a standard PC application. Results of standardized questionnaires showed no significant differences between all application types regarding usability and no significant difference between both VR applications regarding presence. Participants agreed to statements that VR visualizations provide better depth information, using the hands instead of controllers simplifies data exploration, the multisensory VR prototype allows intuitive data exploration, and it is beneficial over traditional data examination methods. While most participants mentioned manual interaction as best aspect, they also found it the most improvable. We conclude that a multisensory XR application with improved manual interaction adds value for volumetric biomedical data examination. We will proceed with our open-source research project ISH3DE (Intuitive Stereoptic Haptic 3D Data Exploration) to serve medical education, therapeutic decisions, surgery preparations, or research data analysis.

Software Engineering,Graphics,Human-Computer Interaction,Image and Video Processing

What problem does this paper attempt to address?

### Problems the Paper Aims to Address

This paper aims to explore whether multisensory extended reality (XR) applications can add value to volumetric biomedical image analysis in medicine and research. Specifically, the paper focuses on the following aspects:

1. **Limitations of Current Visualization Methods**: Although high-resolution volumetric imaging technologies (such as CT and MRI) have become indispensable resources in modern medical diagnosis and treatment, commonly used visualization methods still lag significantly. These methods typically rely on rotating pseudo-3D renderings or artificial fly-throughs on 2D screens, which do not support stereoscopic vision (i.e., perceiving spatial depth through binocular vision).

2. **Need for Multisensory Interaction**: Humans can better perceive the shape and size of 3D objects by integrating tactile and visual information when exploring them. However, existing visualization methods cannot fully utilize this multisensory information.

3. **Challenges of Natural Gesture Interaction**: Although existing commercial XR device controllers can be used for data manipulation, they are not specifically designed for touching and manipulating volumetric data. Therefore, achieving natural gesture interaction remains a challenge.

### Research Objectives

To evaluate the potential value of multisensory XR applications, the researchers conducted the following work:

- **User Testing**: 24 experts who professionally handle volumetric biomedical images in medicine and research were invited to explore the same set of volumetric data using three applications (standard PC application, simple VR prototype, multisensory VR prototype).
- **Questionnaire Survey**: Participants' experiences and opinions on different applications were collected through standardized questionnaires (such as the System Usability Scale (SUS) and the IGroup Presence Questionnaire (IPQ)).

### Key Findings

- **Usability and Presence**: Results from standardized questionnaires showed no significant differences in usability and presence among the three applications.
- **Subjective Evaluation**: The final questionnaire indicated that participants believed multisensory XR applications help better understand the size and shape of organs, simplify data exploration using hand operations compared to VR controllers, allow intuitive data exploration, and may be more beneficial than traditional methods.

### Conclusion

Although current multisensory XR applications still have room for improvement in manual interaction, overall, participants believe that such applications have potential value in the exploration and analysis of volumetric biomedical data. The researchers plan to continue developing their open-source project ISH3DE (Intuitive Stereoptic Haptic 3D Data Exploration) to serve medical education, treatment decision-making, surgical preparation, or research data analysis.