Abstract:Big data techniques have been applied to power grid for the prediction and evaluation of grid conditions. However, the raw data quality can rarely meet the requirement of precise data analytics since raw data set usually contains samples with missing data to which the common data mining models are sensitive. Besides, the raw training data from a single monitoring system, e.g. dissolved gas analysis (DGA), are rarely sufficient for training in the form of valid instances since raw data set usually contains samples with noisy data. Though classic methods like neural network can be used to fill the gaps of missing data and classify the fault type, their models often fail to fit the rules of power grid conditions. This paper presents an integrated data preprocessing framework (DPF) based on Apache Spark to improve the prediction accuracy for data sets with missing data points and classification accuracy with noise data as well as to meet the big data requirement, which mainly combines missing data prediction, data fusion, data cleansing and fault type classification. First, the prediction model is trained based on the linear regression (LinR). Afterwards, we propose an optimized linear method (OLR) to improve the prediction accuracy. Then, to better utilize the strong correlation among different data sources, new data features extracted by persons correlation coefficient (PCC) are fused into a training data set. Next, principal component analysis (PCA) is taken to reduce the side effect brought by the new feature as well as retaining significant information for classification. Finally, the classification model based on logistic regression (LogR) and support vector machine (SVM) is trained to classify the fault type of electric equipment. We test the DPF framework on missing data prediction and fault type classification of power transformers in power grid system. The experimental results show that the predictors based on the proposed framework achieve lower mean square error and the classifiers obtain higher accuracy than traditional ones. Besides, the training time required for training large-scale data shows a decreasing trend. Therefore, the data preprocessing framework DPF would be a good candidate to predict the missing data and classify the fault type in power grid system.

Power Grid Missing Data Filling Method Based on Historical Data Mining Assisted Multi-dimensional Scenario Analysis

Power Demand Combination Forecasting of Rural Power Network Based on Goodness of Fit Empowerment

Research on Association Matching Analysis Technology Between Power Grid Faults and Meteorological Disaster Information

Robust and Automatic Data Cleansing Method for Short-Term Load Forecasting of Distribution Feeders

A Data Fusion and Data Cleaning System for Smart Grids Big Data.

Recovery Algorithm of Power Metering Data Based on Collaborative Fitting

An Integrated Data Preprocessing Framework Based on Apache Spark for Fault Diagnosis of Power Grid Equipment

Study on Missing Data Filling Algorithm of Nuclear Power Plant Operation Parameters

Enhancing Smart Grid Sustainability: Using Advanced Hybrid Machine Learning Techniques While Considering Multiple Influencing Factors for Imputing Missing Electric Load Data

Mat-Transformer-Based State Prediction Method for Information Equipment

A Weekly Load Data Mining Approach Based on Hidden Markov Model

Research on Bad Data Detection in Power Grid Based on Improved Algorithm

Defect Data Mining of Power Consumption Law Based on Improved K-Means Algorithm Clustering

A novel data-characteristic-driven modeling approach for imputing missing value in industrial statistics: A case study of China electricity statistics

Study of Power Grid Development Diagnosis System Based on Multi-Source Data Analysis

Research on Operation Fault Diagnosis Algorithm of Power Grid Equipment Based on Power Big Data

Multimodal Operation Data Mining for Grid Operation Violation Risk Prediction

Missing-Tolerant Anomaly Detection for Gateway Electrical Energy Metering Device Based on Improved Transformer

Intelligent Identification and Order-Sensitive Correction Method of Outliers from Multi-Data Source Based on Historical Data Mining

Data Gap Filling Using Cloud-Based Distributed Markov Chain Cellular Automata Framework for Land Use and Land Cover Change Analysis: Inner Mongolia as a Case Study

Data Repair and Anomaly Detection of Power Meter Based on Deep Reinforcement Learning