Abstract:Big data techniques have been applied to power grid for the prediction and evaluation of grid conditions. However, the raw data quality can rarely meet the requirement of precise data analytics since raw data set usually contains samples with missing data to which the common data mining models are sensitive. Besides, the raw training data from a single monitoring system, e.g. dissolved gas analysis (DGA), are rarely sufficient for training in the form of valid instances since raw data set usually contains samples with noisy data. Though classic methods like neural network can be used to fill the gaps of missing data and classify the fault type, their models often fail to fit the rules of power grid conditions. This paper presents an integrated data preprocessing framework (DPF) based on Apache Spark to improve the prediction accuracy for data sets with missing data points and classification accuracy with noise data as well as to meet the big data requirement, which mainly combines missing data prediction, data fusion, data cleansing and fault type classification. First, the prediction model is trained based on the linear regression (LinR). Afterwards, we propose an optimized linear method (OLR) to improve the prediction accuracy. Then, to better utilize the strong correlation among different data sources, new data features extracted by persons correlation coefficient (PCC) are fused into a training data set. Next, principal component analysis (PCA) is taken to reduce the side effect brought by the new feature as well as retaining significant information for classification. Finally, the classification model based on logistic regression (LogR) and support vector machine (SVM) is trained to classify the fault type of electric equipment. We test the DPF framework on missing data prediction and fault type classification of power transformers in power grid system. The experimental results show that the predictors based on the proposed framework achieve lower mean square error and the classifiers obtain higher accuracy than traditional ones. Besides, the training time required for training large-scale data shows a decreasing trend. Therefore, the data preprocessing framework DPF would be a good candidate to predict the missing data and classify the fault type in power grid system.

An Improved Machine Learning Scheme for Data-Driven Fault Diagnosis of Power Grid Equipment

Deep Learning Neural Network For Power System Fault Diagnosis

Dynamic PCA-based Fault Detection and Diagnosis Analysis

A Methodology to Diagnose Transformer Faults Based on Principal Components Analysis and Artificial Neural Network

Stacked Sparse Autoencoder with PCA and SVM for Data-Based Line Trip Fault Diagnosis in Power Systems

Applied Sensor Fault Detection and Validation Using Transposed Input Data PCA and ANNs.

A Novel Integrated Method to Diagnose Faults in Power Transformers

Novel Approach of Fault Diagnosis in Power Grid Using Information Fusion

An Integrated Data Preprocessing Framework Based on Apache Spark for Fault Diagnosis of Power Grid Equipment

Automatic Online Partial Discharge Diagnosis via Deep Learning

Research on Fault Diagnosis of Power Equipment Based on Big Data

Application of Deep Neural Networks for Fault Diagnosis in a Hybrid AC/DC Power Grid

BA-PNN-based methods for power transformer fault diagnosis

Research on data-driven model for power grid fault diagnosis fusing topological quantification information

Fault Diagnosis of Power Transformers Based on Comprehensive Machine Learning of Dissolved Gas Analysis

Study of a Fault Diagnosis Approach for Power Grid with Information Fusion Based on Multi-Data Resources

Enhancing DGA Transformer Fault Diagnosis Using Data Reconstruction and GD-AHBA-SVM Model

Power Grid Fault Diagnosis Based on Deep Pyramid Convolutional Neural Network

Comprehensive analysis and evaluation of big data for main transformer equipment based on PCA and Apriority

Grounding Grids Fault Diagnosis Based on PCA-BP Neural Network

An Enhanced Semi-Supervised Learning Method with Self-Supervised and Adaptive Threshold for Fault Detection and Classification in Urban Power Grids