Abstract:Software engineering techniques are increasingly relying on deep learning approaches to support many software engineering tasks, from bug triaging to code generation. To assess the efficacy of such techniques researchers typically perform controlled experiments. Conducting these experiments, however, is particularly challenging given the complexity of the space of variables involved, from specialized and intricate architectures and algorithms to a large number of training hyper-parameters and choices of evolving datasets, all compounded by how rapidly the machine learning technology is advancing, and the inherent sources of randomness in the training process. In this work we conduct a mapping study, examining 194 experiments with techniques that rely on deep neural networks appearing in 55 papers published in premier software engineering venues to provide a characterization of the state-of-the-practice, pinpointing experiments common trends and pitfalls. Our study reveals that most of the experiments, including those that have received ACM artifact badges, have fundamental limitations that raise doubts about the reliability of their findings. More specifically, we find: weak analyses to determine that there is a true relationship between independent and dependent variables (87% of the experiments); limited control over the space of DNN relevant variables, which can render a relationship between dependent variables and treatments that may not be causal but rather correlational (100% of the experiments); and lack of specificity in terms of what are the DNN variables and their values utilized in the experiments (86% of the experiments) to define the treatments being applied, which makes it unclear whether the techniques designed are the ones being assessed, or how the sources of extraneous variation are controlled. We provide some practical recommendations to address these limitations.

The Prevalence of Errors in Machine Learning Experiments

How to avoid machine learning pitfalls: a guide for academic researchers

Machine Learning in Environmental Research: Common Pitfalls and Best Practices.

Frequent Errors in Modeling by Machine Learning: A Prototype Case of Predicting the Timely Evolution of COVID-19 Pandemic

Correcting User Decisions Based on Incorrect Machine Learning Decisions

Hazards in Deep Learning Testing: Prevalence, Impact and Recommendations

Questionable practices in machine learning

Validity Concerns of Using Machine Learning in Management Research

Open science perspectives on machine learning for the identification of careless responding: A new hope or phantom menace?

Towards Inferential Reproducibility of Machine Learning Research

Effects of machine learning errors on human decision-making: manipulations of model accuracy, error types, and error importance

Fairness, Accuracy, and Unreliable Data

Making Early Predictions of the Accuracy of Machine Learning Applications

Sources of Irreproducibility in Machine Learning: A Review

Pitfalls in Experiments with DNN4SE: An Analysis of the State of the Practice

Evaluation Gaps in Machine Learning Practice

The Challenges of Machine Learning for Trust and Safety: A Case Study on Misinformation Detection

Common Flaws in Running Human Evaluation Experiments in NLP

A Common Misassumption in Online Experiments with Machine Learning Models

Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinking

Impact of Agents’ Errors on Performance, Reliance and Trust in Human-Agent Collaboration