Abstract:As deep learning systems have been widely deployed in many safety-critical scenarios, their quality and reliability have raised growing concerns. Assuring the quality and evaluating the accuracy of deep learning models could be challenging because, unlike traditional software, DL systems rely on large amounts of labeled data for training and evaluation. The DL models have variability in their behavioral features on datasets with different distributions. In practical application, the potential distribution shift between training and usage scenarios may have an impact on the performance of the model and bring extra vulnerability to DL systems. Although some neuron coverage testing criteria have been proposed to assist in testing the DL systems, they are still limited by the amount of labeled data. Meanwhile, manual labeling test data collected from real-world application scenarios is very time-consuming and costly. In this paper, we propose a novel testing metric, namely LRD, to evaluate the practical accuracy of deep learning systems without requiring the ground truth of test data. The metric uses optimal transport theory to compare model behavior on real-world test data to that on training and out-of-distribution (OOD) sets, by extracting latent representations from the model during input data processing and constructing representation patterns based on the training dataset. The paper further introduces two algorithms powered by the latent representation for out-of-distribution (OOD) data detection and LRD-guided test selection for model retraining. The experimental results show that the evaluation results of LRD have a significant positive correlation with the actual accuracy of the model, and the proposed algorithms are more effective than related OOD detection and test prioritization techniques.

Label-free Evaluation for Performance of Fault Diagnosis Model on Unknown Distribution Dataset

A Novel Imbalanced Data Classification Method Based on Weakly Supervised Learning for Fault Diagnosis

A Real-Time Adaptive Fault Diagnosis Scheme for Dynamic Systems with Performance Degradation

Robust Mechanical Fault Diagnosis with Noisy Label Based on Multistage True Label Distribution Learning.

Are Labels Always Necessary for Classifier Accuracy Evaluation?

A Transfer Learning Method Using High-Quality Pseudo Labels for Bearing Fault Diagnosis

Data-driven fault diagnosis analysis and open-set classification of time-series data

Practical Accuracy Evaluation for Deep Learning Systems Via Latent Representation Discrepancy.

Online Fault Diagnosis in Industrial Processes Using Multimodel Exponential Discriminant Analysis Algorithm

A novel imbalance fault diagnosis method based on data augmentation and hybrid deep learning models

Dual Contrastive Learning for Semi-Supervised Fault Diagnosis Under Extremely Low Label Rate

Benchmarking deep learning models for bearing fault diagnosis using the CWRU dataset: A multi-label approach

A new transferable bearing fault diagnosis method with adaptive manifold probability distribution under different working conditions

A psychological evaluation method incorporating noisy label correction mechanism

Online Multi-Label Classification under Noisy and Changing Label Distribution

SRTFD: Scalable Real-Time Fault Diagnosis through Online Continual Learning

Online Fault Prognosis with Relative Deviation Analysis and Vector Autoregressive Modeling

Imbalanced Real-Time Fault Diagnosis Based on Minority-Prioritized Online Semi-Supervised Random Vector Functional Link Network

Cumulative Probability Distribution Model for Evaluating User Behavior Prediction Algorithms

Energy-based Automated Model Evaluation

A fault diagnosis method for wind turbines gearbox based on adaptive loss weighted meta-ResNet under noisy labels