Abstract:Psychological science has thrived thanks to new methods and innovative practices. Journals, including Behavior Research Methods (BRM), continue to support the dissemination and evaluation of research assets including data, software/hardware, statistical code, and databases of stimuli. However, such research assets rarely allow for computational reproducibility, meaning they are difficult to reuse. Therefore, in this preregistered report, we explore how BRM's authors and BRM structures shape the landscape of functional research assets. Our broad research questions concern: (1) How quickly methods and analytical techniques reported in BRM can be used and developed further by other scientists; (2) Whether functionality has improved following changes to BRM journal policy in support of computational reproducibility; (3) Whether we can disentangle such policy changes from changes in reproducibility over time. We randomly sampled equal numbers of papers (N = 204) published in BRM before and after the implementation of policy changes. Pairs of researchers recorded how long it took to ensure assets (data, software/hardware, statistical code, and materials) were fully operational. They also coded the completeness and reusability of the assets. While improvements were observed in all measures, only changes to completeness were altered significantly following the policy changes (d = .37). The effects varied between different types of research assets, with data sets from surveys/experiments showing the largest improvements in completeness and reusability. Perhaps more importantly, changes to policy do appear to have improved the life span of research products by reducing natural decline. We conclude with a discussion of how, in the future, research and policy might better support computational reproducibility within and beyond psychological science.

Interdisciplinary lessons and recommendations for the evaluation of replicability in behavioral sciences

A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Research into evidence-based psychological interventions needs a stronger focus on replicability

Estimating the reproducibility of psychological science

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Assessing computational reproducibility in Behavior Research Methods

Replication and Reproducibility in Cross-Cultural Psychology

Predicting and reasoning about replicability using structured groups

Replication is important for educational psychology: Recent developments and key issues

Replication marketplaces would help science to become more self-correcting

Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication

Building a Culture of Reproducibility in Academic Research

Are replications mainstream now? A comparison of support for replications expressed in the policies of social psychology journals in 2015 and 2022

Improving Accountability in Recommender Systems Research Through Reproducibility

Registered reports and replications: An ongoing Journal of School Psychology initiative

Replication in music psychology

Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices

Hits and misses in the last decade of open science: Researchers from different subfields and career stages offer personal reflections and suggestions

The reproducibility of statistical results in psychological research: An investigation using unpublished raw data.

ReproduceMe: Lessons from a pilot project on computational reproducibility

Large Scale Replication Projects in Contemporary Psychological Research