Abstract:Large language models trained on source code can support a variety of software development tasks, such as code recommendation and program repair. Large amounts of data for training such models benefit the models' performance. However, the size of the data and models results in long training times and high energy consumption. While publishing source code allows for replicability, users need to repeat the expensive training process if models are not shared. The main goal of the study is to investigate if publications that trained language models for software engineering (SE) tasks share source code and trained artifacts. The second goal is to analyze the transparency on training energy usage. We perform a snowballing-based literature search to find publications on language models for source code, and analyze their reusability from a sustainability standpoint. From 494 unique publications, we identified 293 relevant publications that use language models to address code-related tasks. Among them, 27% (79 out of 293) make artifacts available for reuse. This can be in the form of tools or IDE plugins designed for specific tasks or task-agnostic models that can be fine-tuned for a variety of downstream tasks. Moreover, we collect insights on the hardware used for model training, as well as training time, which together determine the energy consumption of the development process. We find that there are deficiencies in the sharing of information and artifacts for current studies on source code models for software engineering tasks, with 40% of the surveyed papers not sharing source code or trained artifacts. We recommend the sharing of source code as well as trained artifacts, to enable sustainable reproducibility. Moreover, comprehensive information on training times and hardware configurations should be shared for transparency on a model's carbon footprint.

The rewards of reusable machine learning code

Code reuse in gene expression programming

What is the Impact of Releasing Code with Publications? Statistics from the Machine Learning, Robotics, and Control Communities

Toward Reusable Science with Readable Code and Reproducibility

A large-scale study on research code quality and execution

The Landscape of Data Reuse in Interactive Information Retrieval: Motivations, Sources, and Evaluation of Reusability

Code replicability in computer graphics

Building a Culture of Reproducibility in Academic Research

Reproducible Research: Best Practices and Potential Misuse

Building reproducible, reusable, and robust machine learning software

An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code

Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions

Publishing Identifiable Experiment Code And Configuration Is Important, Good and Easy

A Step Toward Quantifying Independently Reproducible Machine Learning Research

Built to Last? Reproducibility and Reusability of Deep Learning Algorithms in Computational Pathology

Better Code, Better Sharing

Large-Scale Code Reuse in Open Source Software

Code Duplication and Reuse in Jupyter Notebooks

On Code Reuse from StackOverflow: an Exploratory Study on Jupyter Notebook

Dataset: Copy-based Reuse in Open Source Software

Better Code, Better Sharing:On the Need of Analyzing Jupyter Notebooks