Abstract:Unit testing has become an essential practice during software development and maintenance. Effective unit tests can help guard and improve software quality but require a substantial amount of time and effort to write and maintain. A unit test consists of a test prefix and a test oracle. Synthesizing test oracles, especially functional oracles, is a well-known challenging problem. Recent studies proposed to leverage neural models to generate test oracles, i.e., neural test oracle generation (NTOG), and obtained promising results. However, after a systematic inspection, we find there are some inappropriate settings in existing evaluation methods for NTOG. These settings could mislead the understanding of existing NTOG approaches’ performance. We summarize them as 1) generating test prefixes from bug-fixed program versions, 2) evaluating with an unrealistic metric, and 3) lacking a straightforward baseline. In this paper, we first investigate the impacts of these settings on evaluating and understanding the performance of NTOG approaches. We find that 1) unrealistically generating test prefixes from bug-fixed program versions inflates the number of bugs found by the state-of-the-art NTOG approach TOGA by 61.8%, 2) FPR (False Positive Rate) is not a realistic evaluation metric and the Precision of TOGA is only 0.38%, and 3) a straightforward baseline NoException, which simply expects no exception should be raised, can find 61% of the bugs found by TOGA with twice the Precision. Furthermore, we introduce an additional ranking step to existing evaluation methods and propose an evaluation metric named Found@K to better measure the cost-effectiveness of NTOG approaches in terms of bug-finding. We propose a novel unsupervised ranking method to instantiate this ranking step, significantly improving the cost-effectiveness of TOGA. Eventually, based on our experimental results and observations, we propose a more realistic evaluation method TEval+ for NTOG and summarize seven rules of thumb to boost NTOG approaches into their practical usages.

A Study of Test Oracle for Application Interface Testing of Distributed System

Towards More Realistic Evaluation for Neural Test Oracle Generation

Automatic Generation of Run-Time Test Oracles for Distributed Real-Time Systems

First, Debug the Test Oracle

Chapter One - Automated Test Oracles: A Survey

Asit: an Interface-oriented Distributed Automated Test System

The Method of Test Oracle Generation Based on Event Temporal Logic

Semantic-Based Test Oracles

DecTest: A Decentralised Testing Architecture for Improving Data Accuracy of Blockchain Oracle

The proposal of a novel software testing framework

Supporting Oracle Construction Via Static Analysis

IMPLEMENTATION AND APPLICATION OF DISTRIBUTED TEST METHOD IN PROTOCOL TESTING

STUDY AND REALIZATION OF A NEW SOFTWARE AUTOMATED TESTING SYSTEM BASED ON J2EE

Oracle Blockades the Application in Trying out the Systematic Design of Technology

Research Of Testing For Scientific Computing Software In The Area Of Nuclear Power Based On Metamorphic Testing

An optimized method for automatic test oracle generation from real-time specification

Novel Software Automated Testing System Based on J2EE

Go-Oracle: Automated Test Oracle for Go Concurrency Bugs

Design of Efficient and Simple Interface Testing Equipment for Opto-Electric Tracking System

Research Advances in Interaction Testing

Survey of Metamorphic Testing