Abstract:Compiler testing is crucially important as compiler is the fundamental infrastructure in software development. One common compiler testing practice leverages a random program generator such as Csmith to generate a huge number of test programs to stress-test compilers. Each of the test programs will be compiled to different executables at different optimization levels and then their outputs will be compared against each other to differentially test compilers. However, the execution time of different test programs varies a lot. Therefore, in practice, developers often set a time limit, such as 60 or 300 seconds, to control the execution of different executables. If the execution exceeds the time limit, it will be terminated. Nevertheless, it is still unclear which time limit is more suitable for compiler testing in this context. We therefore perform the first empirical analysis to investigate how different time limits afffect the efficiency of compiler testing. We found that a time limit of 0.1 seconds can achieve the maximum benefits for compiler testing with the randomly generated test programs. At the same time, we found that 12% test programs requires more than 300 seconds for the execution and these test programs with long-execution-time (LET) consume more than 90% of the entire testing resources which makes compiler testing not cost-effectiveness. We thus propose a framework named ELECT to automatically identify and exclude LET test programs for boosting compiler testing. Our extensive experiments on two popular compilers GCC and LLVM have shown that ELECT can significantly improve the cost-effectiveness of compiler testing as it can respectively detect about 19% and 10% more bugs than the two baseline approaches under the same testing time budget. Besides, ELECT can respectively save about 12% and 38% time than the two other approaches for detecting the same number of bugs.

Testing the Unknown: A Framework for OpenMP Testing via Random Program Generation

Enhancing Performance of Random Testing Through Markov Chain Monte Carlo Methods

An Empirical Comparison Of Compiler Testing Techniques

Guiding Random Test Generation with Program Analysis.

A Divergence-Oriented Approach to Adaptive Random Testing of Java Programs

Random Test Program Generator for Microprocessor Verification

CI/CD Efforts for Validation, Verification and Benchmarking OpenMP Implementations

Boosting Compiler Testing Via Eliminating Test Programs with Long-Execution-Time

Design and Implementation of Random Test Program Generator

Uncertainty-Driven Black-Box Test Data Generation

Using Random Test Selection to Gain Confidence in Modified Software

GRT: an Automated Test Generator Using Orchestrated Program Analysis

An Empirical Comparison of Automated Generation and Classification Techniques for Object-Oriented Unit Testing

Adaptive Random Testing by Exclusion Through Test Profile

Hunting for bugs in code coverage tools via randomized differential testing

Global optimization test problems based on random field composition

An Introduction to Mrandom 3.0

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

Adaptive Test Generation with Qgrams

Adaptive Random Test Case Generation for Combinatorial Testing

Effective Random Test Generation for Deep Learning Compilers