Abstract:Deep neural networks (DNNs) often accept high-dimensional media data (e.g., photos, text, and audio) and understand their perceptual content (e.g., a cat). To test DNNs, diverse inputs are needed to trigger mis-predictions. Some preliminary works use byte-level mutations or domain-specific filters (e.g., foggy), whose enabled mutations may be limited and likely error-prone. SOTA works employ deep generative models to generate (infinite) inputs. Also, to keep the mutated inputs perceptually valid (e.g., a cat remains a "cat" after mutation), existing efforts rely on imprecise and less generalizable heuristics. This study revisits two key objectives in media input mutation - perception diversity (DIV) and validity (VAL) - in a rigorous manner based on manifold, a well-developed theory capturing perceptions of high-dimensional media data in a low-dimensional space. We show important results that DIV and VAL inextricably bound each other, and prove that SOTA generative model-based methods fundamentally fail to mutate real-world media data (either sacrificing DIV or VAL). In contrast, we discuss the feasibility of mutating real-world media data with provably high DIV and VAL based on manifold. We concretize the technical solution of mutating media data of various formats (images, audios, text) via a unified manner based on manifold. Specifically, when media data are projected into a low-dimensional manifold, the data can be mutated by walking on the manifold with certain directions and step sizes. When contrasted with the input data, the mutated data exhibit encouraging DIV in the perceptual traits (e.g., lying vs. standing dog) while retaining reasonably high VAL (i.e., a dog remains a dog). We implement our techniques in DEEPWALK for testing DNNs. DEEPWALK outperforms prior methods in testing comprehensiveness and can find more error-triggering inputs with higher quality.

MuNN: Mutation Analysis of Neural Networks

DeepMutation: Mutation Testing of Deep Learning Systems

There is Limited Correlation Between Coverage and Robustness for Deep Neural Networks

An Empirical Study on Correlation between Coverage and Robustness for Deep Neural Networks

Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing

Mutation Testing of Unsupervised Learning Systems

What Are We Really Testing in Mutation Testing for Machine Learning? A Critical Reflection

DevMuT: Testing Deep Learning Framework Via Developer Expertise-Based Mutation

Mutation Operator Reduction for Deep Learning System

A Probabilistic Framework for Mutation Testing in Deep Neural Networks

MutateNN: Mutation Testing of Image Recognition Models Deployed on Hardware Accelerators

TauMed: Test Augmentation of Deep Learning in Medical Diagnosis

Mutation-Based Adversarial Attacks on Neural Text Detectors

ALPHAGMUT: A Rationale-Guided Alpha Shape Graph Neural Network to Evaluate Mutation Effects

Provably Valid and Diverse Mutations of Real-World Media Data for DNN Testing

Sensitivity Analysis of Deep Neural Networks

Boundary Sampling to Boost Mutation Testing for Deep Learning Models.

BDMMT: Backdoor Sample Detection for Language Models through Model Mutation Testing

Detecting Adversarial Samples with Graph-Guided Testing

DVTest: Deep Neural Network Visualization Testing Framework

Mutation is all you need