Abstract:Current state-of-the-art automatic software repair (ASR) techniques rely heavily on incomplete specifications, or test suites, to generate repairs. This, however, may cause ASR tools to generate repairs that are incorrect and hard to generalize. To assess patch correctness, researchers have been following two methods separately: (1) Automated annotation, wherein patches are automatically labeled by an independent test suite (ITS) - a patch passing the ITS is regarded as correct or generalizable, and incorrect otherwise, (2) Author annotation, wherein authors of ASR techniques manually annotate the correctness labels of patches generated by their and competing tools. While automated annotation cannot ascertain that a patch is actually correct, author annotation is prone to subjectivity. This concern has caused an on-going debate on the appropriate ways to assess the effectiveness of numerous ASR techniques proposed recently. In this work, we propose to assess reliability of author and automated annotations on patch correctness assessment. We do this by first constructing a gold set of correctness labels for 189 randomly selected patches generated by 8 state-of-the-art ASR techniques through a user study involving 35 professional developers as independent annotators. By measuring inter-rater agreement as a proxy for annotation quality - as commonly done in the literature - we demonstrate that our constructed gold set is on par with other high-quality gold sets. We then compare labels generated by author and automated annotations with this gold set to assess reliability of the patch assessment methodologies. We subsequently report several findings and highlight implications for future studies.

Corrections of the NIST Statistical Test Suite for Randomness

A Recoverable Stress Testing Algorithm for Compression and Encryption Cards

On Reliability of Patch Correctness Assessment.

Recommendations on Statistical Randomness Test Batteries for Cryptographic Purposes

Independent Randomness Tests based on the Orthogonalized Non-overlapping Template Matching Test

Statistical testing of random number generators and their improvement using randomness extraction

On Accuracy of Testing Decryption Failure Rate for Encryption Schemes under the LWE Assumption

Deriving the Variance of the Discrete Fourier Transform Test Using Parseval's Theorem

Fiddling the Twiddle Constants - Fault Injection Analysis of the Number Theoretic Transform.

Using Random Test Selection to Gain Confidence in Modified Software

A Taxonomic Review of Adaptive Random Testing: Current Status, Classifications, and Issues

Building Test Batteries Based on Analyzing Random Number Generator Tests within the Framework of Algorithmic Information Theory

Consistency of invariance-based randomization tests

Study on upper limit of sample sizes for a two-level test in NIST SP800-22

On-line Anomaly Detection and Qualification of Random Bit Streams

Probabilistic Models for Estimation of Random and Pseudo-Random Test Length

Efficient Method for Random Fault Attack Against Aes Hardware Implementation

A Systematic Approach and Analysis of Key Mismatch Attacks on Lattice-Based NIST Candidate KEMs

Corrected Evaluation Results of the NTCIR WWW-2, WWW-3, and WWW-4 English Subtasks

Testing equality of spectral densities using randomization techniques

Correcting Subverted Random Oracles