Abstract:Unit testing has become an essential practice during software development and maintenance. Effective unit tests can help guard and improve software quality but require a substantial amount of time and effort to write and maintain. A unit test consists of a test prefix and a test oracle. Synthesizing test oracles, especially functional oracles, is a well-known challenging problem. Recent studies proposed to leverage neural models to generate test oracles, i.e., neural test oracle generation (NTOG), and obtained promising results. However, after a systematic inspection, we find there are some inappropriate settings in existing evaluation methods for NTOG. These settings could mislead the understanding of existing NTOG approaches’ performance. We summarize them as 1) generating test prefixes from bug-fixed program versions, 2) evaluating with an unrealistic metric, and 3) lacking a straightforward baseline. In this paper, we first investigate the impacts of these settings on evaluating and understanding the performance of NTOG approaches. We find that 1) unrealistically generating test prefixes from bug-fixed program versions inflates the number of bugs found by the state-of-the-art NTOG approach TOGA by 61.8%, 2) FPR (False Positive Rate) is not a realistic evaluation metric and the Precision of TOGA is only 0.38%, and 3) a straightforward baseline NoException, which simply expects no exception should be raised, can find 61% of the bugs found by TOGA with twice the Precision. Furthermore, we introduce an additional ranking step to existing evaluation methods and propose an evaluation metric named Found@K to better measure the cost-effectiveness of NTOG approaches in terms of bug-finding. We propose a novel unsupervised ranking method to instantiate this ranking step, significantly improving the cost-effectiveness of TOGA. Eventually, based on our experimental results and observations, we propose a more realistic evaluation method TEval+ for NTOG and summarize seven rules of thumb to boost NTOG approaches into their practical usages.

An Effective Method for Identifying Unknown Unknowns with Noisy Oracle.

Towards More Realistic Evaluation for Neural Test Oracle Generation

A Semantic Labeling Strategy to Reject Unknown Objects in Large Scale 3d Point Clouds

A Probabilistic Model of Active Learning with Multiple Noisy Oracles

Uncertainty-Aware Learning against Label Noise on Imbalanced Datasets

Unknown Sniffer for Object Detection: Don't Turn a Blind Eye to Unknown Objects

Unknown Identity Rejection Loss: Utilizing Unlabeled Data for Face Recognition

Exploratory Machine Learning with Unknown Unknowns

The research on an algorithm for identification of OE models with missing input data

Application of Conditional Random Fields Model in Unknown Words Identification

USD: Unknown Sensitive Detector Empowered by Decoupled Objectness and Segment Anything Model

Which Strategies Matter for Noisy Label Classification? Insight into Loss and Uncertainty

Uncertainty-inspired open set learning for retinal anomaly identification

SVDD-based outlier detection on uncertain data

Online Algorithms with Uncertainty-Quantified Predictions

A Survey on Uncertainty Quantification Methods for Deep Learning

Uncertainty-guided label correction with wavelet-transformed discriminative representation enhancement

ALIM: Adjusting Label Importance Mechanism for Noisy Partial Label Learning

Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real World