Abstract:Software engineering techniques are increasingly relying on deep learning approaches to support many software engineering tasks, from bug triaging to code generation. To assess the efficacy of such techniques researchers typically perform controlled experiments. Conducting these experiments, however, is particularly challenging given the complexity of the space of variables involved, from specialized and intricate architectures and algorithms to a large number of training hyper-parameters and choices of evolving datasets, all compounded by how rapidly the machine learning technology is advancing, and the inherent sources of randomness in the training process. In this work we conduct a mapping study, examining 194 experiments with techniques that rely on deep neural networks appearing in 55 papers published in premier software engineering venues to provide a characterization of the state-of-the-practice, pinpointing experiments common trends and pitfalls. Our study reveals that most of the experiments, including those that have received ACM artifact badges, have fundamental limitations that raise doubts about the reliability of their findings. More specifically, we find: weak analyses to determine that there is a true relationship between independent and dependent variables (87% of the experiments); limited control over the space of DNN relevant variables, which can render a relationship between dependent variables and treatments that may not be causal but rather correlational (100% of the experiments); and lack of specificity in terms of what are the DNN variables and their values utilized in the experiments (86% of the experiments) to define the treatments being applied, which makes it unclear whether the techniques designed are the ones being assessed, or how the sources of extraneous variation are controlled. We provide some practical recommendations to address these limitations.

Against naïve induction from experimental data

Induction and Learning

Assume a can opener

Inferencing into the void: problems with implicit populations Comments on `Empirical software engineering experts on the use of students and professionals in experiments'

Invited Commentary: The Need for Cognitive Science in Methodology

Time to Stop and Think: What kind of research do we want to do?

Beyond integrative experiment design: Systematic experimentation guided by causal discovery AI

Inferential statistics, power estimates, and study design formalities continue to suppress biomedical innovation

In Defense of Experimental Consumer Psychology

The same analysis approach: Practical protection against the pitfalls of novel neuroimaging analysis methods

Keep it real: rethinking the primacy of experimental control in cognitive neuroscience

Don't let perfect be the enemy of better: In defense of unparameterized megastudies

Experiment Selection in Meta-Analytic Piecemeal Causal Discovery

Does Meta-induction Justify Induction: Or Maybe Something Else?

Replies to commentaries on beyond playing 20 questions with nature

Induction, Popper, and machine learning

The contingent animal: does artificial innateness misrepresent behavioral development?

Pitfalls in Experiments with DNN4SE: An Analysis of the State of the Practice

Towards Futuristic Autonomous Experimentation--A Surprise-Reacting Sequential Experiment Policy

Tantalus on the Road to Asymptopia

Eliminativist induction cannot be a solution to psychology's crisis