Abstract:Unit testing has become an essential practice during software development and maintenance. Effective unit tests can help guard and improve software quality but require a substantial amount of time and effort to write and maintain. A unit test consists of a test prefix and a test oracle. Synthesizing test oracles, especially functional oracles, is a well-known challenging problem. Recent studies proposed to leverage neural models to generate test oracles, i.e., neural test oracle generation (NTOG), and obtained promising results. However, after a systematic inspection, we find there are some inappropriate settings in existing evaluation methods for NTOG. These settings could mislead the understanding of existing NTOG approaches’ performance. We summarize them as 1) generating test prefixes from bug-fixed program versions, 2) evaluating with an unrealistic metric, and 3) lacking a straightforward baseline. In this paper, we first investigate the impacts of these settings on evaluating and understanding the performance of NTOG approaches. We find that 1) unrealistically generating test prefixes from bug-fixed program versions inflates the number of bugs found by the state-of-the-art NTOG approach TOGA by 61.8%, 2) FPR (False Positive Rate) is not a realistic evaluation metric and the Precision of TOGA is only 0.38%, and 3) a straightforward baseline NoException, which simply expects no exception should be raised, can find 61% of the bugs found by TOGA with twice the Precision. Furthermore, we introduce an additional ranking step to existing evaluation methods and propose an evaluation metric named Found@K to better measure the cost-effectiveness of NTOG approaches in terms of bug-finding. We propose a novel unsupervised ranking method to instantiate this ranking step, significantly improving the cost-effectiveness of TOGA. Eventually, based on our experimental results and observations, we propose a more realistic evaluation method TEval+ for NTOG and summarize seven rules of thumb to boost NTOG approaches into their practical usages.

Automatic Test Oracle for Image Processing Applications Using Support Vector Machines

Towards More Realistic Evaluation for Neural Test Oracle Generation

Self-Checked Metamorphic Testing of an Image Processing Program

Using Metamorphic Relations to Verify and Enhance Artcode Classification

First, Debug the Test Oracle

Computer Vision Intelligence Test Modeling and Generation: A Case Study on Smart OCR

Semantic-Based Test Oracles

The Computer Vision-based Tolerancing Callout Detection Model

Neural-Based Test Oracle Generation: A Large-scale Evaluation and Lessons Learned

Automated Testing of Image Captioning Systems

TOGLL: Correct and Strong Test Oracle Generation with LLMs

Learning to Encode and Classify Test Executions

Human Visual System-Based Fundus Image Quality Assessment of Portable Fundus Camera Photographs.

ATOM: Automated Black-Box Testing of Multi-Label Image Classification Systems.

Using Machine Learning to Generate Test Oracles: A Systematic Literature Review

Saliency-Based Defect Detection in Industrial Images by Using Phase Spectrum

Identifying implementation bugs in machine learning based image classifiers using metamorphic testing

Metamorphic Testing of Image Captioning Systems via Image-Level Reduction

Testing and Validating Machine Learning Classifiers by Metamorphic Testing.

Augmenting automatically generated unit-test suites with regression oracle checking

Machines that test Software like Humans