Experiments as Code and its application to VR studies in human-building interaction

Leonel Aguilar,Michal Gath-Morad,Jascha Grübel,Jasper Ermatinger,Hantao Zhao,Stefan Wehrli,Robert W. Sumner,Ce Zhang,Dirk Helbing,Christoph Hölscher

DOI: https://doi.org/10.1038/s41598-024-60791-3

IF: 4.6

2024-05-02

Scientific Reports

Abstract:Experiments as Code (ExaC) is a concept for reproducible, auditable, debuggable, reusable, & scalable experiments. Experiments are a crucial tool to understand Human-Building Interactions (HBI) and build a coherent theory around it. However, a common concern for experiments is their auditability and reproducibility. Experiments are usually designed, provisioned, managed, and analyzed by diverse teams of specialists (e.g., researchers, technicians, engineers) and may require many resources (e.g., cloud infrastructure, specialized equipment). Although researchers strive to document experiments accurately, this process is often lacking. Consequently, it is difficult to reproduce these experiments. Moreover, when it is necessary to create a similar experiment, the "wheel is very often reinvented". It appears easier to start from scratch than trying to reuse existing work. Thus valuable embedded best practices and previous experiences are lost. In behavioral studies, such as in HBI, this has contributed to the reproducibility crisis. To tackle these challenges, we propose the ExaC paradigm, which not only documents the whole experiment, but additionally provides the automation code to provision, deploy, manage, and analyze the experiment. To this end, we define the ExaC concept, provide a taxonomy for the components of a practical implementation, and provide a proof of concept with an HBI desktop VR experiment that demonstrates the benefits of its "as code" representation, that is, reproducibility, auditability, debuggability, reusability, & scalability.

multidisciplinary sciences

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to improve the repeatability, auditability, debuggability, reusability and extensibility of virtual reality (VR) experiments in the field of human - building interaction (HBI). Specifically, the author points out that the current VR behavioral experiments face the following challenges:

1. **Audit and repeatability issues**: Existing VR experiments are difficult to audit and repeat because the experiment design, configuration and analysis are usually completed by different expert teams and lack detailed documentation. This leads to the fact that slight differences in the experimental process may affect the reproducibility of the results.

2. **Waste of resources**: Due to the lack of an effective sharing mechanism, researchers often need to design similar experiments from scratch, resulting in the waste of precious best practices and experiences.

3. **Crisis of scientific repeatability**: Especially in behavioral research, many attempts to replicate previous experiments have failed, raising concerns about the reliability of research results.

To solve these problems, the author proposes the concept of "Experiments as Code" (ExaC), that is, representing all aspects of the experiment (including components, procedures and documents) in the form of structured and executable code. In this way, the deployment, management and analysis processes of the experiment can be automated, thus ensuring the transparency and consistency of the experiment.

### Specific goals of ExaC

1. **Improve repeatability**: By providing complete automated code and digital documents, other researchers can easily reproduce the experiment.
2. **Enhance auditability**: Ensure that third parties can comprehensively evaluate the design and execution details of the experiment.
3. **Improve debuggability**: Facilitate the identification and correction of factors affecting the experimental results.
4. **Promote reuse**: Allow researchers to reuse some or all components of existing experiments to create new experiments.
5. **Support expansion**: Enable the experiment to be carried out on a larger scale, for example, by increasing the number of participants or using a remote experimental platform.

### Implementation methods

The author defines a six - pillar taxonomy (see Figure 2), covering the key aspects of the experiment: documentation, infrastructure and environment, data collection, data analysis, data assembly and management. Based on this taxonomy, the author constructs a prototype implementation of a desktop VR experiment, demonstrating the application potential of ExaC in the HBI field.

### Main contributions

1. **Propose and define the ExaC paradigm** and demonstrate its application through a complete desktop VR behavioral experiment.
2. **Provide an open - source ExaC implementation**, allowing third parties to review and reuse the experiment design and verify the advantages of ExaC.
3. **Compare and review existing VR frameworks**, highlighting their advantages and disadvantages in terms of supporting experiments, experimental processes and pre - productivity.

Through these efforts, the author aims to provide a more reliable and efficient research tool for VR experiments in the HBI field to meet the current challenges of scientific repeatability.