Abstract:In recent years, deep learning (DL) applications have been widely used in both industrial and academic domains. Bugs in the DL framework have become one of the leading causes of DI. model training and deployment failures. At present, fuzzing has emerged as one of the most effective and commonly used techniques to detect bugs in the DL framework. However, existing fuzzing techniques focus on testing the high-level APIs and use a random-based generation approach to explore the vast input space of API parameters. This kind of approach suffers low effectiveness because it cannot distinguish which lines in the lower-level component have been executed. We analyzed the structure of the most popular DL frameworks (e.g., TensorFlow and PyTorch) and found that line coverage information from the Python layer can be used to characterize the execution of the underlying layers. Therefore, we propose PCFuzz, a Python coverage -guided fuzzing technique, to address the above question. The main idea of PCFuzz is to perform guided fuzzing tests based on the Python line coverage of lower-level components. To collect Python coverage at a low cost, we designed a light weighted Python instrumentation approach specifically for DL frameworks. On the basis of this instrumentation, PCFuzz then leverages coverage to optimize the mutation scheduling, thereby enhancing the overall effectiveness of DL framework fuzzing. We conduct experiments on the most popular DL frameworks TensorFlow and PyTorch and compare PCFuzz with the state-of-the-art model -level fuzzer LEMON and API -level fuzzer FreeFuzz. The results show that PCFuzz is more efficient in covering the codes in the low-level Python libraries and raw operations. Specifically, within the same time cost, PCFuzz achieves on average 13.9x higher Python line coverage than LEMON. For FreeFuzz, PCFuzz achieves on average 52.56% higher Python line coverage. In addition, during our preliminary experiments, PCFuzz discovered 1 previously unknown hug in TensorFlow.

Python Coverage Guided Fuzzing for Deep Learning Framework

DeepCov: Coverage Guided Deep Learning Framework Fuzzing

A First Look at the Effect of Deep Learning in Coverage-guided Fuzzing

FA-Fuzz: A Novel Scheduling Scheme Using Firefly Algorithm for Mutation-Based Fuzzing

Coverage-guided fuzzing for deep reinforcement learning systems

The Seeds of the FUTURE Sprout from History: Fuzzing for Unveiling Vulnerabilities in Prospective Deep-Learning Libraries

Deep Learning Framework Fuzzing Based on Model Mutation

Python Fuzzing for Trustworthy Machine Learning Frameworks

DRLFCfuzzer: fuzzing with Deep-Reinforcement-Learning under Format Constraints

DeFuzz: Deep Learning Guided Directed Fuzzing

DLFuzz: Differential Fuzzing Testing of Deep Learning Systems.

FDFuzz: Applying Feature Detection to Fuzz Deep Learning Systems

Lightweight Code Coverage Analysis for Deep Learning Framework Testing

Graph-Based Fuzz Testing for Deep Learning Inference Engine

Graph-based Fuzz Testing for Deep Learning Inference Engines

A Lightweight and High-Precision Approach for Bulky JavaScript Engines Fuzzing

Coverage Guided Differential Adversarial Testing of Deep Learning Systems

Context-Aware Fuzzing for Robustness Enhancement of Deep Learning Models

LAFuzz: Neural Network for Efficient Fuzzing

Toward Understanding Deep Learning Framework Bugs

Deep Learning-Based Hybrid Fuzz Testing