Abstract:Deep learning (DL) applications are prevalent nowadays as they can help with multiple tasks. DL libraries are essential for building DL applications. Furthermore, DL operators are the important building blocks of the DL libraries, that compute the multi-dimensional data (tensors). Therefore, bugs in DL operators can have great impacts. Testing is a practical approach for detecting bugs in DL operators. In order to test DL operators effectively, it is essential that the test cases pass the input validity check and are able to reach the core function logic of the operators. Hence, extracting the input validation constraints is required for generating high-quality test cases. Existing techniques rely on either human effort or documentation of DL library APIs to extract the constraints. They cannot extract complex constraints and the extracted constraints may differ from the actual code implementation. To address the challenge, we propose ACETest, a technique to automatically extract input validation constraints from the code to build valid yet diverse test cases which can effectively unveil bugs in the core function logic of DL operators. For this purpose, ACETest can automatically identify the input validation code in DL operators, extract the related constraints and generate test cases according to the constraints. The experimental results on popular DL libraries, TensorFlow and PyTorch, demonstrate that ACETest can extract constraints with higher quality than state-of-the-art (SOTA) techniques. Moreover, ACETest is capable of extracting 96.4% more constraints and detecting 1.95 to 55 times more bugs than SOTA techniques. In total, we have used ACETest to detect 108 previously unknown bugs on TensorFlow and PyTorch, with 87 of them confirmed by the developers. Lastly, five of the bugs were assigned with CVE IDs due to their security impacts.

Automatic testing of runtime enforcers with Test4Enforcers

An Experience-Based Approach for Test Execution Effort Estimation

Correct-by-Construction Runtime Enforcement in AI -- A Survey

Automatic Web Testing using Curiosity-Driven Reinforcement Learning

An Autonomous Performance Testing Framework using Self-Adaptive Fuzzy Reinforcement Learning

Guardian: A Runtime Framework for LLM-based UI Exploration

ACETest: Automated Constraint Extraction for Testing Deep Learning Operators

Automated Behavioral Regression Testing

Test Agents: Adaptive, Autonomous and Intelligent Test Cases

Effective testing of Android apps using extended IFML models

Beyond Tests

MOTEF: A Testing Framework for Runtime Monitoring Infrastructures

Industrial Control Systems Security via Runtime Enforcement

Runtime enforcement of reactive systems using synchronous enforcers

Automatically Detecting Exception Handling Defects in Android Applications.

Alleviating Patch Overfitting with Automatic Test Generation: A Study of Feasibility and Effectiveness for the Nopol Repair System

Towards Scalable Automated Mobile App Testing

A Model and Implementation of Function Test Automation for Web Applications

Runtime Enforcement for Component-Based Systems

Leveraging Android Automated Testing to Assist Crowdsourced Testing

Engineering Trustworthy Self-Adaptive Software with Dynamic Assurance Cases