Abstract:In recent years, outliers caused by manual operation errors and equipment acquisition failures often occur, bringing challenges to big data analysis. In view of the difficulties in identifying and correcting outliers of multi-source data, an intelligent identification and order-sensitive correction method of outliers from multi-data sources based on historical data mining was proposed. First, an intelligent identification method of outliers of single-source data is proposed based on neural tangent kernel K-means (NTKKM) clustering. The original data is mapped to high-dimensional feature space using Neural Tangent Kernel, where the features of outliers are acquired by K-means clustering to realize the accurate identification of outliers. Second, an order-sensitive missing value imputation framework for multi-source data (OMSMVI) was proposed. The similarity graph of sources with missing data was constructed based on multidimensional similarity analysis, and the filling order decision was transformed into an optimization problem to realize the optimal filling order decision of missing values in multi-source data. Finally, a neighborhood-based imputation (NI) algorithm is proposed. Based on the traditional KNN filling algorithm, neighboring nodes of sources with missing data are flexibly selected to the achieve accurate correction of outliers. The case experiment was operated on actual power grid data, and the results show that the proposed clustering method can identify outliers more accurately, and the determined optimal imputation sequence has higher accuracy, which provide a feasible new idea for the identification and correction of outliers in the process of data preprocessing.

Integration Of Ann And Statistical Method For Outlier Detection In Complex System

Robust Principal Component Analysis Based on Maximum Correntropy Criterion

Outlier Detection Algorithm Based on Reachable Neighbor

A Novel Nonlinear Dynamic System Modeling Approach Using Radial Basis Function Neural Networks

A Novel Bayesian Robust Model and Its Application for Fault Detection and Automatic Supervision of Nonlinear Process

A Multivariable Statistical Process Monitoring Method Based On Multiscale Analysis And Principal Curves

Outlier Detection Method Based on Compressed Time Series and Voronoi Diagram for Power Data

Similarity Based Robust Probability Latent Variable Regression Model and Its Kernel Extension for Process Monitoring

A method for outlier detection based on cluster analysis and visual expert criteria

Nonlinear Process Monitoring Using Improved Kernel Principal Component Analysis

Study on Test Method for Outlier of Flight Data

A neighborhood weighted-based method for the detection of outliers

Comparative Study of Neighbor-based Methods for Local Outlier Detection

Robust and Sparse Kernel PCA and Its Outlier Map

Outlier detection based on improved SOM and its application in power system

An Iterative Algorithm for Robust Kernel Principal Component Analysis

Intelligent Identification and Order-Sensitive Correction Method of Outliers from Multi-Data Source Based on Historical Data Mining

Unsupervised Bayesian change-point detection approach for reliable prognostics and health management of complex mechanical systems

Robust and Sparse Principal Component Analysis With Adaptive Loss Minimization for Feature Selection

The Outlier Detection Algorithm and Its Application in the Statistical Monitoring Model Based on Modified Scaling