Abstract:In order to improve the data quality, the big data cleaning method of distribution network was studied in this paper. First, the Local Outlier Factor (LOF) algorithm based on DBSCAN clustering was used to detect outliers. However, due to the difficulty in determining the LOF threshold, a method of dynamically calculating the threshold based on the transformer districts and time was proposed. Besides, the LOF algorithm combines the statistical distribution method to reduce the "misjudgment rate". Aiming at the diversity and complexity of data missing forms in power big data, this paper improved the Random Forest imputation algorithm, which can be applied to various forms of missing data, especially the blocked missing data and even some horizontal or vertical data completely missing. The data in this paper were from real data of 44 transformer districts of a certain 10kV line in distribution network. Experimental results showed that outlier detection was accurate and suitable for any shape and multidimensional power big data. The improved Random Forest imputation algorithm was suitable for all missing forms, with higher imputation accuracy and better model stability. By comparing the network loss prediction between the data using this data cleaning method and the data removing outliers and missing values, it was found that the accuracy of network loss prediction had been improved by nearly 4 percentage points using the data cleaning method mentioned in this paper. Additionally, as the proportion of bad data increased, the difference between the prediction accuracy of cleaned data and that of uncleaned data was greater.

Fast Cleaning Method for Smart Grid Data Based on Sparse Self-Coding

A Data Fusion and Data Cleaning System for Smart Grids Big Data.

Robust and Automatic Data Cleansing Method for Short-Term Load Forecasting of Distribution Feeders

The Data Cleaning of Electric Industrial Control Terminal Based on the Iforest and Genetic BP Neural Network Algorithms

Electric Load Data Compression and Classification Based on Deep Stacked Auto-Encoders

Big data cleaning model of smart grid based on Tensor Tucker decomposition

Cleaning method for abnormal energy big data based on sparse self-coding

Big Data Cleaning Model of Multi-Source Heterogeneous Power Grid Based On Machine Learning Classification Algorithm

Safety Monitoring of Power Industrial Control Terminals Based on Data Cleaning

Grid-Constrained Data Cleansing Method for Enhanced Busload Forecasting

Cleaning Method for Big Data of Power Transmission and Transformation Equipment State Based on Time Sequence Analysis

Akane: Perplexity-Guided Time Series Data Cleaning

An intelligent scheduling control method for smart grid based on deep learning

A big data cleaning method based on improved CLOF and Random Forest for distribution network

Sparse and Redundant Representation-Based Smart Meter Data Compression and Pattern Extraction

Data Cleaning Based on Multi-sensor Spatiotemporal Correlation

Cleaning of Abnormal Wind Speed Power Data Based on Quartile RANSAC Regression

A fast method for identifying bad data of massive power network data

Online Monitoring Data Cleaning of Transformer Considering Time Series Correlation

Data-driven smart grid carbon emission control methods using graph-based power flow computing

Smart Grid Time Series Big Data Processing System