Abstract:Psychological science has thrived thanks to new methods and innovative practices. Journals, including Behavior Research Methods (BRM), continue to support the dissemination and evaluation of research assets including data, software/hardware, statistical code, and databases of stimuli. However, such research assets rarely allow for computational reproducibility, meaning they are difficult to reuse. Therefore, in this preregistered report, we explore how BRM's authors and BRM structures shape the landscape of functional research assets. Our broad research questions concern: (1) How quickly methods and analytical techniques reported in BRM can be used and developed further by other scientists; (2) Whether functionality has improved following changes to BRM journal policy in support of computational reproducibility; (3) Whether we can disentangle such policy changes from changes in reproducibility over time. We randomly sampled equal numbers of papers (N = 204) published in BRM before and after the implementation of policy changes. Pairs of researchers recorded how long it took to ensure assets (data, software/hardware, statistical code, and materials) were fully operational. They also coded the completeness and reusability of the assets. While improvements were observed in all measures, only changes to completeness were altered significantly following the policy changes (d = .37). The effects varied between different types of research assets, with data sets from surveys/experiments showing the largest improvements in completeness and reusability. Perhaps more importantly, changes to policy do appear to have improved the life span of research products by reducing natural decline. We conclude with a discussion of how, in the future, research and policy might better support computational reproducibility within and beyond psychological science.

Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and Reuse

Open Science Practices by Early Career HCI Researchers: Perceptions, Challenges, and Benefits

What and How Are We Reporting in HRI? A Review and Recommendations for Reporting Recruitment, Compensation, and Gender

Lessons learned: A neuroimaging research center's transition to open and reproducible science

A standardized framework to test event-based experiments

Reproducibility of in-vivo electrophysiological measurements in mice

An illustration of reproducibility in neuroscience research in the absence of selective reporting

Scanning the horizon: towards transparent and reproducible neuroimaging research

Double trouble? The communication dimension of the reproducibility crisis in experimental psychology and neuroscience

Open and reproducible neuroimaging: From study inception to publication

Interactive Tools for Reproducible Science -- Understanding, Supporting, and Motivating Reproducible Science Practices

Supporting the Design and Analysis of HCI Experiments

Reproducibility in Human-Robot Interaction: Furthering the Science of HRI

Risk or Chance? Large Language Models and Reproducibility in HCI Research

A Systematic Review on Reproducibility in Child-Robot Interaction

Assessing computational reproducibility in Behavior Research Methods

Reducing the Human Factor in Virtual Reality Research to Increase Reproducibility and Replicability

On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices

Sharing Neurophysiology Data from the Allen Brain Observatory: Lessons Learned

A New Research Trend in Social Neuroscience: Towards an Interactive-Brain Neuroscience

Reproducibility in the Absence of Selective Reporting: an Illustration from Large-Scale Brain Asymmetry Research