Abstract:Reproducibility of results is, in all research fields, the cornerstone of the scientific method and the minimum standard for assessing the value of scientific claims and conclusions drawn by other scientists. It requires a systematic approach and accurate description of the experimental procedure and data analysis, which allows other scientists to follow the steps described in the published work and obtain the "same results." In general and in different research contexts with "same" results, we mean different things. It can be almost identical measures in a fully deterministic experiment or "validation of a hypothesis" or statistically similar results in a non-deterministic context. Unfortunately, it has been shown by systematic meta-analysis studies that many findings in fields like psychology, sociology, medicine, and economics do not hold up when other researchers try to replicate them. Many scientific fields are experiencing what is generally referred to as a "reproducibility crisis," which undermines the trust in published results, imposes a thorough revision of the methodology in scientific research, and makes progress difficult. In general, the reproducibility of experiments is not a mainstream practice in artificial intelligence and robotics research. Surgical robotics is no exception. There is a need for developing new tools and putting in place a community effort to allow the transition to more reproducible research and hence faster progress in research. Reproducibility, replicability, and benchmarking (operational procedures for the assessment and comparison of research results) are made more complex for medical robotics and surgical systems, due to patenting, safety, and ethical issues. In this review paper, we selected 10 relevant published manuscripts on surgical robotics to analyze their clinical applicability and underline the problems related to reproducibility of the reported experiments, with the aim of finding possible solutions to the challenges that limit the translation of many scientific research studies into real-world applications and slow down research progress.

Reproducibility in Human-Robot Interaction: Furthering the Science of HRI

A Systematic Review on Reproducibility in Child-Robot Interaction

What and How Are We Reporting in HRI? A Review and Recommendations for Reporting Recruitment, Compensation, and Gender

Reducing the Human Factor in Virtual Reality Research to Increase Reproducibility and Replicability

RISE: an open-source architecture for interdisciplinary and reproducible human-robot interaction research

Human Robot Interaction: Applications, Challenges and Future Directions

Ethical Challenges in the Human-Robot Interaction Field

15 Years of (Who)man Robot Interaction: Reviewing the H in Human-Robot Interaction

Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and Reuse

Reproducibility challenges in robotic surgery

Reproducibility in Research: Systems, Infrastructure, Culture

Human-Robot Interaction in Groups: Methodological and Research Practices

Studying robots outside the lab: HRI as ethnography

Transferability of HRI Research: Potential and Challenges

Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues

Human-Robot Collaboration: From Psychology to Social Robotics

Towards Formalizing HRI Data Collection Processes

Proceedings of the AI-HRI Symposium at AAAI-FSS 2018

Reproducibility in Machine Learning-Driven Research

Proceedings of the AI-HRI Symposium at AAAI-FSS 2020

Interactive Tools for Reproducible Science -- Understanding, Supporting, and Motivating Reproducible Science Practices