Abstract:Big data techniques have been applied to power grid for the prediction and evaluation of grid conditions. However, the raw data quality can rarely meet the requirement of precise data analytics since raw data set usually contains samples with missing data to which the common data mining models are sensitive. Besides, the raw training data from a single monitoring system, e.g. dissolved gas analysis (DGA), are rarely sufficient for training in the form of valid instances since raw data set usually contains samples with noisy data. Though classic methods like neural network can be used to fill the gaps of missing data and classify the fault type, their models often fail to fit the rules of power grid conditions. This paper presents an integrated data preprocessing framework (DPF) based on Apache Spark to improve the prediction accuracy for data sets with missing data points and classification accuracy with noise data as well as to meet the big data requirement, which mainly combines missing data prediction, data fusion, data cleansing and fault type classification. First, the prediction model is trained based on the linear regression (LinR). Afterwards, we propose an optimized linear method (OLR) to improve the prediction accuracy. Then, to better utilize the strong correlation among different data sources, new data features extracted by persons correlation coefficient (PCC) are fused into a training data set. Next, principal component analysis (PCA) is taken to reduce the side effect brought by the new feature as well as retaining significant information for classification. Finally, the classification model based on logistic regression (LogR) and support vector machine (SVM) is trained to classify the fault type of electric equipment. We test the DPF framework on missing data prediction and fault type classification of power transformers in power grid system. The experimental results show that the predictors based on the proposed framework achieve lower mean square error and the classifiers obtain higher accuracy than traditional ones. Besides, the training time required for training large-scale data shows a decreasing trend. Therefore, the data preprocessing framework DPF would be a good candidate to predict the missing data and classify the fault type in power grid system.

The Data Cleaning of Electric Industrial Control Terminal Based on the Iforest and Genetic BP Neural Network Algorithms

Safety Monitoring of Power Industrial Control Terminals Based on Data Cleaning

Robust and Automatic Data Cleansing Method for Short-Term Load Forecasting of Distribution Feeders

Non-intrusive Runtime Monitoring for Power System Intelligent Terminal Based on Improved Deep Belief Networks (I-DBN)

An Anomaly Detection Algorithm for the Power Industrial Terminal Security Monitoring

A Data Fusion and Data Cleaning System for Smart Grids Big Data.

An Improved Data-Efficiency Algorithm Based on Combining Isolation Forest and Mean Shift for Anomaly Data Filtering in Wind Power Curve

Research on Deep Detection Technology of Abnormal Behavior of Power Industrial Control System

Electricity Consumption Forecasting Based on Improved BP Neural Network

Data Cleaning for Power Quality Monitoring

Big Data Cleaning Model of Multi-Source Heterogeneous Power Grid Based On Machine Learning Classification Algorithm

Theft of Electricity Detection Based on Supervised Learning

Deep Learning Neural Network For Power System Fault Diagnosis

Grid-Constrained Data Cleansing Method for Enhanced Busload Forecasting

Optimizing the Isolation Forest Algorithm for Identifying Abnormal Behaviors of Students in Education Management Big Data

Anomaly detection of power grid dispatching platform based on Isolation Forest and K-means fusion algorithm

Application of BP neural network in electricity users classification

A graph neural network-based data cleaning method to prevent intelligent fault diagnosis from data contamination

Data cleaning method for the process of acid production with flue gas based on improved random forest

Anomaly Detection for Power Consumption Data Based on Isolated Forest

An Integrated Data Preprocessing Framework Based on Apache Spark for Fault Diagnosis of Power Grid Equipment