Abstract:In recent years, the research community has raised serious questions about the reproducibility of scientific work. In particular, since many studies include some kind of computing work, reproducibility is also a technological challenge, not only in computer science, but in most research domains. Replicability and computational reproducibility are not easy to achieve, not only because researchers have diverse proficiency in computing technologies, but also because of the variety of computational environments that can be used. Indeed, it is challenging to recreate the same environment using the same frameworks, code, data sources, programming languages, dependencies, and so on. In this work, we propose an Integrated Development Environment allowing the share, configuration, packaging and execution of an experiment by setting the code and data used and defining the programming languages, code, dependencies, databases, or commands to execute to achieve consistent results for each experiment. After the initial creation and configuration, the experiment can be executed any number of times, always producing exactly the same results. Furthermore, it allows the execution of the experiment by using a different associated dataset, and it can be possible to verify the reproducibility and replicability of the results. This allows the creation of a reproducible pack that can be re-executed by anyone on any other computer. Our platform aims to allow researchers in any field to create a reproducibility package for their science that can be re-executed on any other computer. To evaluate our platform, we used it to reproduce 25 experiments extracted from published papers. We have been able to successfully reproduce 20 (80%) of these experiments achieving the results reported in such works with minimum effort, thus showing that our approach is effective.

Towards Sharing Task Environments to Support Reproducible Evaluations of Interactive Recommender Systems

Towards Creating a Standardized Collection of Simple and Targeted Experiments to Analyze Core Aspects of the Recommender Systems Problem

Improving Accountability in Recommender Systems Research Through Reproducibility

Towards Employing Recommender Systems for Supporting Data and Algorithm Sharing

Beyond Collaborative Filtering: A Relook at Task Formulation in Recommender Systems

E Pluribus Unum: Guidelines on Multi-Objective Evaluation of Recommender Systems

A Reproducible Analysis of Sequential Recommender Systems

Reciprocal Recommender Systems: Analysis of State-of-Art Literature, Challenges and Opportunities towards Social Recommendation

"Share and Enjoy": Publishing Useful and Usable Scientific Models

SUBER: An RL Environment with Simulated Human Behavior for Recommender Systems

BARS: Towards Open Benchmarking for Recommender Systems

Sharing Knowledge in Multi-Task Deep Reinforcement Learning

A Collaborative Approach to Computational Reproducibility

Collaborative training of heterogeneous reinforcement learning agents in environments with sparse rewards: what and when to share?

Task-Based Evaluation

RBoard: A Unified Platform for Reproducible and Reusable Recommender System Benchmarks

SARDINE: A Simulator for Automated Recommendation in Dynamic and Interactive Environments

Modelling the Recommender Alignment Problem

Toward Simulating Environments in Reinforcement Learning Based Recommendations.

BETA-Rec: Build, Evaluate and Tune Automated Recommender Systems.

A Backend Platform for Supporting the Reproducibility of Computational Experiments