Abstract:Unit testing has become an essential practice during software development and maintenance. Effective unit tests can help guard and improve software quality but require a substantial amount of time and effort to write and maintain. A unit test consists of a test prefix and a test oracle. Synthesizing test oracles, especially functional oracles, is a well-known challenging problem. Recent studies proposed to leverage neural models to generate test oracles, i.e., neural test oracle generation (NTOG), and obtained promising results. However, after a systematic inspection, we find there are some inappropriate settings in existing evaluation methods for NTOG. These settings could mislead the understanding of existing NTOG approaches’ performance. We summarize them as 1) generating test prefixes from bug-fixed program versions, 2) evaluating with an unrealistic metric, and 3) lacking a straightforward baseline. In this paper, we first investigate the impacts of these settings on evaluating and understanding the performance of NTOG approaches. We find that 1) unrealistically generating test prefixes from bug-fixed program versions inflates the number of bugs found by the state-of-the-art NTOG approach TOGA by 61.8%, 2) FPR (False Positive Rate) is not a realistic evaluation metric and the Precision of TOGA is only 0.38%, and 3) a straightforward baseline NoException, which simply expects no exception should be raised, can find 61% of the bugs found by TOGA with twice the Precision. Furthermore, we introduce an additional ranking step to existing evaluation methods and propose an evaluation metric named Found@K to better measure the cost-effectiveness of NTOG approaches in terms of bug-finding. We propose a novel unsupervised ranking method to instantiate this ranking step, significantly improving the cost-effectiveness of TOGA. Eventually, based on our experimental results and observations, we propose a more realistic evaluation method TEval+ for NTOG and summarize seven rules of thumb to boost NTOG approaches into their practical usages.

Neuroevolution-Based Generation of Tests and Oracles for Games

Combining Neuroevolution with the Search for Novelty to Improve the Generation of Test Inputs for Games

Towards More Realistic Evaluation for Neural Test Oracle Generation

Neural-Based Test Oracle Generation: A Large-scale Evaluation and Lessons Learned

A Neuroevolution Approach to General Atari Game Playing

Using a NEAT approach with curriculums for dynamic content generation in video games

Playing a 2D Game Indefinitely using NEAT and Reinforcement Learning

Machine Learning and Evolutionary Computing for GUI-based Regression Testing

Engaging Young Learners with Testing Using the Code Critters Mutation Game

Automated Test Oracle Based On Neural Networks

An electronic-game framework for evaluating coevolutionary algorithms

Automated test generation for Scratch programs

Using Machine Learning to Generate Test Oracles: A Systematic Literature Review

Neural Task Synthesis for Visual Programming

GAVEL: Generating Games Via Evolution and Language Models

Test Oracle Generation Based on BPNN by Using the Values of Variables at Different Breakpoints for Programs.

Search-based Testing for Scratch Programs

Automatic Test Oracle Based on Probabilistic Neural Networks

Co-generation of game levels and game-playing agents

Generative Adversarial Neuroevolution for Control Behaviour Imitation

Neuroevolution of Recurrent Architectures on Control Tasks