Abstract:Computer science is also an experimental science. This is particularly the case for parallel computing, which is in a total state of flux, and where experiments are necessary to substantiate, complement, and challenge theoretical modeling and analysis. Here, experimental work is as important as are advances in theory, that are indeed often driven by the experimental findings. In parallel computing, scientific contributions presented in research articles are therefore often based on experimental data, with a substantial part devoted to presenting and discussing the experimental findings. As in all of experimental science, experiments must be presented in a way that makes reproduction by other researchers possible, in principle. Despite appearance to the contrary, we contend that reproducibility plays a small role, and is typically not achieved. As can be found, articles often do not have a sufficiently detailed description of their experiments, and do not make available the software used to obtain the claimed results. As a consequence, parallel computational results are most often impossible to reproduce, often questionable, and therefore of little or no scientific value. We believe that the description of how to reproduce findings should play an important part in every serious, experiment-based parallel computing research article. We aim to initiate a discussion of the reproducibility issue in parallel computing, and elaborate on the importance of reproducible research for (1) better and sounder technical/scientific papers, (2) a sounder and more efficient review process and (3) more effective collective work. This paper expresses our current view on the subject and should be read as a position statement for discussion and future work. We do not consider the related (but no less important) issue of the quality of the experimental design.

Challenges and Techniques for Reproducible Simulations

"Can I Implement Your Algorithm?": A Model for Reproducible Research Software

Reproducibility in Research: Systems, Infrastructure, Culture

Case Studies and Challenges in Reproducibility in the Computational Sciences

Bitwise Identical Compiling Setup: Prospective for Reproducibility and Reliability of Earth System Modeling

Practical resources for enhancing the reproducibility of mechanistic modeling in systems biology

Learning from reproducing computational results: introducing three principles and the Reproduction Package

Beyond the Badge: Reproducibility Engineering as a Lifetime Skill

Truth, Proof, and Reproducibility: There's no counter-attack for the codeless

Replicability of simulation studies for the investigation of statistical methods: the RepliSims project

Reproducibility, Replicability, and Repeatability: A survey of reproducible research with a focus on high performance computing

Computational Reproducibility in Computational Social Science

Wanted: standards for automatic reproducibility of computational experiments

Reproducibility of Issues Reported in Stack Overflow Questions: Challenges, Impact & Estimation

A Collaborative Approach to Computational Reproducibility

Reproducibility in Learning

Primer on Reproducible Research in R: Enhancing Transparency and Scientific Rigor

The fundamental principles of reproducibility

On the State and Importance of Reproducible Experimental Research in Parallel Computing

Reproducibility Signals in Science: A preliminary analysis

Reproducibility in the Classroom