Comprehensive Evaluation of Big Data Quality in Power Systems with Entropy Weight and Grey System Theory
Gang LI,Yafei JIAO,Fuyan LIU,Min YU,Yu SONG,Fushuan WEN
DOI: https://doi.org/10.3969/j.issn.1000-7229.2016.12.003
2017-01-01
Abstract:With the continuous expansion of power systems, as well as ever-developing technology and reduced costs of measurement devices, the recorded data in power systems have been increasing significantly and progressively exhibit the feature of big data. Much attention has been paid to the full use of big data for improving the planning, operation and control of power system, and hence how to evaluate the quality of big data is becoming an important problem to be examined. Some research publications are available on data quality improvement, such as data cleaning, data integration, and the detection of similar records, but the existing research work is still preliminary in data quality evaluations. Given this background, considering the characteristics of power systems and associated big data, this paper proposes a comprehensive method for evaluating the quality of big data in power systems. Firstly, we construct an index system for big data quality evaluations. Then aiming at the timeliness of big data, we adopt the K-means clustering algorithm in parallel with MapReduce for fast preprocessing of the big data sample set. Secondly, we use entropy weight method to calculate the objective weight of each dataset and grey evaluation method to determine the data quality level. On this basis, the comprehensive evaluation of the sample data set is carried out. Finally, the recorded electric load historical data in a city power company are employed to demonstrate the proposed method.