Abstract:Deep neural networks (DNNs) are becoming an integral part of most software systems. Previous work has shown that DNNs have bugs. Unfortunately, existing debugging techniques do not support localizing DNN bugs because of the lack of understanding of model behaviors. The entire DNN model appears as a black box. To address these problems, we propose an approach that automatically determines whether the model is buggy or not, and identifies the root causes. Our key insight is that historic trends in values propagated between layers can be analyzed to identify faults, and localize faults. To that end, we first enable dynamic analysis of deep learning applications: by converting it into an imperative representation and alternatively using a callback mechanism. Both mechanisms allows us to insert probes that enable dynamic analysis over the traces produced by the DNN while it is being trained on the training data. We then conduct dynamic analysis over the traces to identify the faulty layer that causes the error. We propose an algorithm for identifying root causes by capturing any numerical error and monitoring the model during training and finding the relevance of every layer on the DNN outcome. We have collected a benchmark containing 40 buggy models and patches that contain real errors in deep learning applications from Stack Overflow and GitHub. Our benchmark can be used to evaluate automated debugging tools and repair techniques. We have evaluated our approach using this DNN bug-and-patch benchmark, and the results showed that our approach is much more effective than the existing debugging approach used in the state of the practice Keras library. For 34 out of 40 cases, our approach was able to detect faults whereas the best debugging approach provided by Keras detected 32 out of 40 faults. Our approach was able to localize 21 out of 40 bugs whereas Keras did not localize any faults.

TFIDF-FL: Localizing Faults Using Term Frequency-Inverse Document Frequency and Deep Learning.

A General Noise-Reduction Framework for Fault Localization of Java Programs.

A Dynamic Fault Localization Technique with Noise Reduction for Java Programs

DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs

ALBFL: A Novel Neural Ranking Model for Software Fault Localization Via Combining Static and Dynamic Features

Fault Localization in Deep Learning-based Software: A System-level Approach

Software Fault Localization Based on Multi-objective Feature Fusion and Deep Learning

Dynamic Data Fault Localization for Deep Neural Networks

Fault Localization with Code Coverage Representation Learning

ABFL: an Autoencoder Based Practical Approach for Software Fault Localization.

An Effective Data-Driven Approach for Localizing Deep Learning Faults

Path Analysis for Effective Fault Localization in Deep Neural Networks

Nonlocal Structure Constrained Neighborhood Preserving Embedding Model and Its Application for Fault Detection

A novel collaborative bearing fault diagnosis method based on multi-scale dynamic fusion network under speed fluctuating condition

Adopting Pre-and Post-processing Weight Mechanisms to Improve Deep Learning-based Fault Localization.

The fault frequency priors fusion deep learning framework with application to fault diagnosis of offshore wind turbines

TFN: An Interpretable Neural Network with Time-Frequency Transform Embedded for Intelligent Fault Diagnosis

Farthest-Nearest Distance Neighborhood and Locality Projections Integrated with Bootstrap for Industrial Process Fault Diagnosis

VsusFL: Variable-suspiciousness-based Fault Localization for novice programs

DeepLocalize: Fault Localization for Deep Neural Networks

A Multiple-Criteria Ensemble Weight Strategy to Increase the Effectiveness of Deep Learning-based Fault Localization