Abstract:In recent years, deep learning (DL) applications have been widely used in both industrial and academic domains. Bugs in the DL framework have become one of the leading causes of DI. model training and deployment failures. At present, fuzzing has emerged as one of the most effective and commonly used techniques to detect bugs in the DL framework. However, existing fuzzing techniques focus on testing the high-level APIs and use a random-based generation approach to explore the vast input space of API parameters. This kind of approach suffers low effectiveness because it cannot distinguish which lines in the lower-level component have been executed. We analyzed the structure of the most popular DL frameworks (e.g., TensorFlow and PyTorch) and found that line coverage information from the Python layer can be used to characterize the execution of the underlying layers. Therefore, we propose PCFuzz, a Python coverage -guided fuzzing technique, to address the above question. The main idea of PCFuzz is to perform guided fuzzing tests based on the Python line coverage of lower-level components. To collect Python coverage at a low cost, we designed a light weighted Python instrumentation approach specifically for DL frameworks. On the basis of this instrumentation, PCFuzz then leverages coverage to optimize the mutation scheduling, thereby enhancing the overall effectiveness of DL framework fuzzing. We conduct experiments on the most popular DL frameworks TensorFlow and PyTorch and compare PCFuzz with the state-of-the-art model -level fuzzer LEMON and API -level fuzzer FreeFuzz. The results show that PCFuzz is more efficient in covering the codes in the low-level Python libraries and raw operations. Specifically, within the same time cost, PCFuzz achieves on average 13.9x higher Python line coverage than LEMON. For FreeFuzz, PCFuzz achieves on average 52.56% higher Python line coverage. In addition, during our preliminary experiments, PCFuzz discovered 1 previously unknown hug in TensorFlow.

Deep Learning Framework Fuzzing Based on Model Mutation

DeepCov: Coverage Guided Deep Learning Framework Fuzzing

Mutation-Based Deep Learning Framework Testing Method in JavaScript Environment

Toward Understanding Deep Learning Framework Bugs

Python Coverage Guided Fuzzing for Deep Learning Framework

Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing

DeepMutation: Mutation Testing of Deep Learning Systems

FA-Fuzz: A Novel Scheduling Scheme Using Firefly Algorithm for Mutation-Based Fuzzing

DevMuT: Testing Deep Learning Framework Via Developer Expertise-Based Mutation

FDFuzz: Applying Feature Detection to Fuzz Deep Learning Systems

The Seeds of the FUTURE Sprout from History: Fuzzing for Unveiling Vulnerabilities in Prospective Deep-Learning Libraries

DRLFCfuzzer: fuzzing with Deep-Reinforcement-Learning under Format Constraints

Audee: Automated Testing for Deep Learning Frameworks

MoCo: Fuzzing Deep Learning Libraries Via Assembling Code

FuzzCoder: Byte-level Fuzzing Test via Large Language Model

Graph-Based Fuzz Testing for Deep Learning Inference Engine

A First Look at the Effect of Deep Learning in Coverage-guided Fuzzing

Graph-based Fuzz Testing for Deep Learning Inference Engines

MalFuzz: Coverage-guided fuzzing on deep learning-based malware classification model

Deep Learning Framework Testing Via Hierarchical and Heuristic Model Generation.

Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models