Abstract:Bug-related research, e.g., fault localization, program repair, and software testing, relies heavily on high-quality and large-scale software bug repositories. The importance of such repositories is twofold. On one side, real-world bugs and their associated patches may inspire novel approaches for finding, locating, and repairing software bugs. On the other side, the real-world bugs and their patches are indispensable for rigorous and meaningful evaluation of approaches to software testing, fault localization, and program repair. To this end, a number of software bug repositories, e.g., iBUGS and Defects4J, have been constructed recently by mining version control systems and bug tracking systems. However, fully automated construction of bug repositories by simply taking bug-fixing commits from version control systems often results in inaccurate patches that contain many bug-irrelevant changes. Although we may request experts or developers to manually exclude the bug-irrelevant changes (as the authors of Defects4J did), such extensive human intervention makes it difficult to build large-scale bug repositories. To this end, in this paper, we propose an automatic approach, called BugBuilder, to construct bug repositories from version control systems. Different from existing approaches, it automatically extracts complete and concise bug-fixing patches and excludes bug-irrelevant changes. It first detects and excludes software refactorings involved in bug-fixing commits. BugBuilder then enumerates all subsets of the remaining part, and discards invalid subsets by compilation and software testing. If exactly a single subset survives the validation, this subset is taken as the complete and concise bug-fixing patch for the associated bug. In case multiple subsets survive, BugBuilder employs a sequence of heuristics to select the most likely one. Evaluation results on 809 real-world bug-fixing commits in Defects4J suggest that BugBuilder successfully extracted complete and concise bug-fixing patches from forty-three percent of the bug-fixing commits, and its precision (99%) was even higher than human experts. We also built a bug repository, called GrowingBugs, with the proposed approach. The resulting repository serves as evidence of the usefulness of the proposed approach, as well as a publicly available benchmark for bug-related research.

GBGallery: A benchmark and framework for game testing

A Unified Framework for Mini-game Testing: Experience on WeChat

An Empirical Study of Bugs in Build Process

GLIB: Towards Automated Test Oracle for Graphically-Rich Applications

BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies

Deriving and Evaluating a Detailed Taxonomy of Game Bugs

GameRTS: A Regression Testing Framework for Video Games.

BugBuilder: an Automated Approach to Building Bug Repository.

BOAT: an Experimental Platform for Researchers to Comparatively and Reproducibly Evaluate Bug Localization Techniques

A Survey of Video Game Testing

BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixes

Tests4Py: A Benchmark for System Testing

PreciseBugCollector: Extensible, Executable and Precise Bug-fix Collection

TrickyBugs: A Dataset of Corner-case Bugs in Plausible Programs

GitBug-Java: A Reproducible Benchmark of Recent Java Bugs

An Empirical Study on Benchmarks of Artificial Software Vulnerabilities

Automata-Based Trace Analysis for Aiding Diagnosing GUI Testing Tools for Android

Automatic Detection of Performance Bugs in Database Systems Using Equivalent Queries

An Empirical Study on Real Bug Fixes

An Empirical Study of Bugs in Software Build System

JaConTeBe: A Benchmark Suite of Real-World Java Concurrency Bugs (T)