Abstract:Context: Mutation Testing (MT) is an important tool in traditional Software Engineering (SE) white-box testing. It aims to artificially inject faults in a system to evaluate a test suite's capability to detect them, assuming that the test suite defects finding capability will then translate to real faults. If MT has long been used in SE, it is only recently that it started gaining the attention of the Deep Learning (DL) community, with researchers adapting it to improve the testability of DL models and improve the trustworthiness of DL systems. Objective: If several techniques have been proposed for MT, most of them neglected the stochasticity inherent to DL resulting from the training phase. Even the latest MT approaches in DL, which propose to tackle MT through a statistical approach, might give inconsistent results. Indeed, as their statistic is based on a fixed set of sampled training instances, it can lead to different results across instances set when results should be consistent for any instance. Methods: In this work, we propose a Probabilistic Mutation Testing (PMT) approach that alleviates the inconsistency problem and allows for a more consistent decision on whether a mutant is killed or not. Results: We show that PMT effectively allows a more consistent and informed decision on mutations through evaluation using three models and eight mutation operators used in previously proposed MT methods. We also analyze the trade-off between the approximation error and the cost of our method, showing that relatively small error can be achieved for a manageable cost. Conclusion: Our results showed the limitation of current MT practices in DNN and the need to rethink them. We believe PMT is the first step in that direction which effectively removes the lack of consistency across test executions of previous methods caused by the stochasticity of DNN training.

Towards Mutation Testing of Reinforcement Learning Systems

Mutation Testing of Reinforcement Learning Systems.

Mutation Testing of Unsupervised Learning Systems

muPRL: A Mutation Testing Pipeline for Deep Reinforcement Learning based on Real Faults

RGChaser: A RL-guided Fuzz and Mutation Testing Framework for Deep Learning Systems

What Are We Really Testing in Mutation Testing for Machine Learning? A Critical Reflection

DeepMutation: Mutation Testing of Deep Learning Systems

Mutation Testing framework for Machine Learning

Reinforcement Learning for Safety Testing: Lessons from A Mobile Robot Case Study

Reinforcement Learning for Mutation Operator Selection in Automated Program Repair

MILE: A Mutation Testing Framework of In-Context Learning Systems

Learning from Mutants: Using Code Mutation to Learn and Monitor Invariants of a Cyber-Physical System

DevMuT: Testing Deep Learning Framework Via Developer Expertise-Based Mutation

Regression mutation testing.

Manifesting Bugs in Machine Learning Code: An Explorative Study with Mutation Testing

Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testing

Test Scenario Generation for Autonomous Driving Systems with Reinforcement Learning

A Probabilistic Framework for Mutation Testing in Deep Neural Networks

Mutation-Based Deep Learning Framework Testing Method in JavaScript Environment

An Exploratory Study on Using Large Language Models for Mutation Testing

Improving Testing of Deep-Learning Systems