Reducing the Human Factor in Virtual Reality Research to Increase Reproducibility and Replicability

Daniel Hepperle,Tobias Dienlin,Matthias Wölfel

DOI: https://doi.org/10.48550/arXiv.2110.15687

2021-10-29

Abstract:The replication crisis is real, and awareness of its existence is growing across disciplines. We argue that research in human-computer interaction (HCI), and especially virtual reality (VR), is vulnerable to similar challenges due to many shared methodologies, theories, and incentive structures. For this reason, in this work, we transfer established solutions from other fields to address the lack of replicability and reproducibility in HCI and VR. We focus on reducing errors resulting from the so-called human factor and adapt established solutions to the specific needs of VR research. In addition, we present a toolkit to support the setup, execution, and evaluation of VR research. Some of the features aim to reduce human errors and thus improve replicability and reproducibility. Finally, the identified chances are applied to a typical scientific process in VR.

Human-Computer Interaction,Methodology

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve reproducibility and replicability in virtual reality (VR) research. Specifically, the author believes that the human factor introduces errors in VR research, and these errors affect the reliability and credibility of research results. To reduce these errors, the paper proposes the following points:

1. **Implementation Standards**: Reduce repetitive work by providing frameworks or toolkits that offer sets of validated methods, such as the implementation of questionnaires to reduce the interruption of presence, and the preparation of data records for further analysis.

2. **Process Documentation**: Clear process documentation is crucial for the reproducibility of research. The paper emphasizes the need for extended documentation beyond the scope of journal standards to help researchers understand different conclusions in the research, thereby formulating hypotheses and reporting results.

3. **Optimizing Generalization Ability**: In traditional research environments, the characteristics of participants and researchers (such as height, race, age) are fixed, while in VR, these characteristics can be freely modified through avatars, allowing for more flexible research designs.

4. **Sharing Test Environments**: Since most of the content in the VR research process is digital, the complete test environment can be saved and shared through online repositories (such as GitHub and GitLab), which helps other researchers reproduce the experiments.

5. **Technical Limitations**: Although VR research devices are usually cheaper than traditional laboratory facilities, researchers still need to invest in appropriate hardware, software, and content. The rapid iterative hardware development also poses challenges to reproducibility and replicability.

The paper also introduces a toolkit named Virtual Reality Scientific Toolkit (VRSTK), which aims to support the setup, execution, and evaluation of VR research. This toolkit reduces redundant work and improves the transparency and reliability of research by providing predefined scenarios, real - time data recording, and export functions.

In conclusion, the goal of this paper is to improve the reproducibility and replicability of VR research by reducing errors caused by the human factor, thereby enhancing the credibility and application value of research results.