Abstract:Big data techniques have been applied to power grid for the prediction and evaluation of grid conditions. However, the raw data quality can rarely meet the requirement of precise data analytics since raw data set usually contains samples with missing data to which the common data mining models are sensitive. Besides, the raw training data from a single monitoring system, e.g. dissolved gas analysis (DGA), are rarely sufficient for training in the form of valid instances since raw data set usually contains samples with noisy data. Though classic methods like neural network can be used to fill the gaps of missing data and classify the fault type, their models often fail to fit the rules of power grid conditions. This paper presents an integrated data preprocessing framework (DPF) based on Apache Spark to improve the prediction accuracy for data sets with missing data points and classification accuracy with noise data as well as to meet the big data requirement, which mainly combines missing data prediction, data fusion, data cleansing and fault type classification. First, the prediction model is trained based on the linear regression (LinR). Afterwards, we propose an optimized linear method (OLR) to improve the prediction accuracy. Then, to better utilize the strong correlation among different data sources, new data features extracted by persons correlation coefficient (PCC) are fused into a training data set. Next, principal component analysis (PCA) is taken to reduce the side effect brought by the new feature as well as retaining significant information for classification. Finally, the classification model based on logistic regression (LogR) and support vector machine (SVM) is trained to classify the fault type of electric equipment. We test the DPF framework on missing data prediction and fault type classification of power transformers in power grid system. The experimental results show that the predictors based on the proposed framework achieve lower mean square error and the classifiers obtain higher accuracy than traditional ones. Besides, the training time required for training large-scale data shows a decreasing trend. Therefore, the data preprocessing framework DPF would be a good candidate to predict the missing data and classify the fault type in power grid system.

Data-driven Solutions for Power System Fault Analysis and Novelty Detection.

Anomaly Detection Algorithm Based on Electric Equipment

Data Mining And Analysis Of Tree-Caused Faults In Power Distribution Systems

Novel Approach of Fault Diagnosis in Power Grid Using Information Fusion

Deep Learning Neural Network For Power System Fault Diagnosis

Biologically Inspired Intelligent Fault Diagnosis for Power Distribution Systems

Data Mining Applications to Fault Diagnosis in Power Electronic Systems: A Systematic Review

Application of Data Mining Method based on Deep Learning in Power System

A Review of Data Driven-based Incipient Fault Diagnosis

Distributed Data Mining For Root Causes Of Kpi Faults In Wireless Networks

Application of Data-mining Based on MAS to the Failure Diagnosis in Power Supply

An Integrated Data Preprocessing Framework Based on Apache Spark for Fault Diagnosis of Power Grid Equipment

Data Mining and Machine Learning for Power System Monitoring, Understanding, and Impact Evaluation

Probability-based Fault Detection Method and Its Learning Algorithm

Data-mining Massive Real-Time Data in a Power Plant: Challenges, Problems and Solutions

Early Warning of Power Plant Equipment Based on Massive Real-Time Data Mining Technology

Applications of data mining time series to power systems disturbance analysis

An Identification Method for Anomaly Types of Active Distribution Network Based on Data Mining

A Novel Arc Fault Detection Method Integrated Random Forest, Improved Multi-scale Permutation Entropy and Wavelet Packet Transform

Data Analysis and Algorithm Innovation in Power System IntelligentMonitoring and Early Warning Technology

Data-Driven and Interactive Fault Diagnosis of Distribution Switches