Abstract:Big data techniques have been applied to power grid for the prediction and evaluation of grid conditions. However, the raw data quality can rarely meet the requirement of precise data analytics since raw data set usually contains samples with missing data to which the common data mining models are sensitive. Besides, the raw training data from a single monitoring system, e.g. dissolved gas analysis (DGA), are rarely sufficient for training in the form of valid instances since raw data set usually contains samples with noisy data. Though classic methods like neural network can be used to fill the gaps of missing data and classify the fault type, their models often fail to fit the rules of power grid conditions. This paper presents an integrated data preprocessing framework (DPF) based on Apache Spark to improve the prediction accuracy for data sets with missing data points and classification accuracy with noise data as well as to meet the big data requirement, which mainly combines missing data prediction, data fusion, data cleansing and fault type classification. First, the prediction model is trained based on the linear regression (LinR). Afterwards, we propose an optimized linear method (OLR) to improve the prediction accuracy. Then, to better utilize the strong correlation among different data sources, new data features extracted by persons correlation coefficient (PCC) are fused into a training data set. Next, principal component analysis (PCA) is taken to reduce the side effect brought by the new feature as well as retaining significant information for classification. Finally, the classification model based on logistic regression (LogR) and support vector machine (SVM) is trained to classify the fault type of electric equipment. We test the DPF framework on missing data prediction and fault type classification of power transformers in power grid system. The experimental results show that the predictors based on the proposed framework achieve lower mean square error and the classifiers obtain higher accuracy than traditional ones. Besides, the training time required for training large-scale data shows a decreasing trend. Therefore, the data preprocessing framework DPF would be a good candidate to predict the missing data and classify the fault type in power grid system.

The DeMaDs Open Source Modeling Framework for Power System Malfunction Detection

An Operational Data-Driven Malfunction Detection Framework for Enhanced Power Distribution System Monitoring -- The DeMaDs Approach

Automated Data-Driven Model Extraction and Validation of Inverter Dynamics with Grid Support Function

Simulation of Multi-Stage Attack and Defense Mechanisms in Smart Grids

Deep Machine Learning Model-Based Cyber-Attacks Detection in Smart Power Systems

Supporting Future Electrical Utilities: Using Deep Learning Methods in EMS and DMS Algorithms

Model checking-based decision support system for fault management: A comprehensive framework and application in electric power systems

Smart Grid Anomaly Detection using a Deep Learning Digital Twin

An Integrated Data Preprocessing Framework Based on Apache Spark for Fault Diagnosis of Power Grid Equipment

A Framework for Synthetic Power System Dynamics

Online Model-Free Cyber Attack Detection in Smart Grid Using Dynamic Mode Decomposition

Feasibility Study of Neural ODE and DAE Modules for Power System Dynamic Component Modeling

PowerModelsRestoration.jl: An open-source framework for exploring power network restoration algorithms

Intelligent Fault Analysis in Electrical Power Grids

Parallel Statistical Model Checking for Safety Verification in Smart Grids

System modeling based on machine learning for anomaly detection and predictive maintenance in industrial plants

Improving Power Grid Monitoring Data Quality: an Efficient Machine Learning Framework for Missing Data Prediction

An Integrated Online Dynamic Security Assessment System for Improved Situational Awareness and Economic Operation

Statistical Structure Learning, Towards a Robust Smart Grid

Sneak Preview: PowerDynamics.jl -= An Open-Source library for analyzing dynamic stability in power grids with high shares of renewable energy

Big Data Analysis Based on Fault Detection Modeling of Microgrid Information Flow