Abstract:Software engineering techniques are increasingly relying on deep learning approaches to support many software engineering tasks, from bug triaging to code generation. To assess the efficacy of such techniques researchers typically perform controlled experiments. Conducting these experiments, however, is particularly challenging given the complexity of the space of variables involved, from specialized and intricate architectures and algorithms to a large number of training hyper-parameters and choices of evolving datasets, all compounded by how rapidly the machine learning technology is advancing, and the inherent sources of randomness in the training process. In this work we conduct a mapping study, examining 194 experiments with techniques that rely on deep neural networks appearing in 55 papers published in premier software engineering venues to provide a characterization of the state-of-the-practice, pinpointing experiments common trends and pitfalls. Our study reveals that most of the experiments, including those that have received ACM artifact badges, have fundamental limitations that raise doubts about the reliability of their findings. More specifically, we find: weak analyses to determine that there is a true relationship between independent and dependent variables (87% of the experiments); limited control over the space of DNN relevant variables, which can render a relationship between dependent variables and treatments that may not be causal but rather correlational (100% of the experiments); and lack of specificity in terms of what are the DNN variables and their values utilized in the experiments (86% of the experiments) to define the treatments being applied, which makes it unclear whether the techniques designed are the ones being assessed, or how the sources of extraneous variation are controlled. We provide some practical recommendations to address these limitations.

Recommendations for analysing and meta-analysing small sample size software engineering experiments

Effect sizes and their variance for AB/BA crossover design studies

Moving Beyond the Mean: Analyzing Variance in Software Engineering Experiments

Using Monte Carlo experiments to select meta‐analytic estimators

Focus on Data: Statistical Design of Experiments and Sample Size Selection Using Power Analysis

Small Studies May Overestimate the Effect Sizes in Critical Care Meta-Analyses: A Meta-Epidemiological Study

Simulation-Based Power Analyses for the Smallest Effect Size of Interest: A Confidence-Interval Approach for Minimum-Effect and Equivalence Testing

Determining Sample Size in Binary Measurement System

Estimating the sample mean and standard deviation from commonly reported quantiles in meta-analysis

Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances

Estimating the sample mean and standard deviation from order statistics and sample size in meta-analysis

Evaluating Small Sample Approaches for Model Test Statistics in Structural Equation Modeling

Publication Bias: A Detailed Analysis of Experiments Published in ESEM

Sample size requirements for stated choice experiments

Sampling in software engineering research: a critical review and guidelines

A simple statistical framework for small sample studies

Pitfalls in Experiments with DNN4SE: An Analysis of the State of the Practice

Meta-analysis Technique and its Application in Software Engineering

Inference at Scale Significance Testing for Large Search and Recommendation Experiments

Apples, Oranges, and Software Engineering: Study Selection Challenges for Secondary Research on Latent Variables

"Medium-n studies" in computing education conferences