Abstract:With the rapid development of information technology and internet, all kinds of industry data exploded causing difficult to analyze and mine useful information from big data. Traditional analysis system has bottlenecks of performance and scalability in big data processing. The research and development of novel and efficient big data analysis and mining platform has become the focus of all organizations. Along with the development of smart grid, power data with characteristics of power industry needs more targeted and efficient data mining analysis. In this paper, aiming at the shortage of existing work, we propose a distributed big data mining platform based on distributed system infrastructure such as Hadoop and Spark. The platform develops and implements a variety of rapid highly parallel mining algorithm by Spark and Tensorflow, including machine learning, statistics and analysis, deep learning and so on. Using the OSGI technology to build low coupling component model, the platform improve reusability of component algorithm, introduces the workflow engine and user-friendly GUI, reduces the complexity of the user operations, support user-defined data mining tasks. For the characteristics of smart grid big data, the platform develops and improves the dozens of algorithm components about data processing and analysis. And designing a scalable algorithms library and the component library greatly improves the scalability of big data mining platform and processing smart grid data. Our platform has already been launched in a state grid Company, satisfying the demand of various smart grid data analysis business.

Data-mining Massive Real-Time Data in a Power Plant: Challenges, Problems and Solutions

Use of Data Mining Techniques in Process Control and Optimization of Thermal Power Plant

The research and application of data mining in power plant operation optimization

Early Warning of Power Plant Equipment Based on Massive Real-Time Data Mining Technology

Big Data Platform for Smart Substation Monitoring System

An Uniform Access Method of Heterogeneous Big Data with Power Grid Application

The Data Mining Under the Dig Data Era and Its Application in Electric Power Enterprises

Smart Grid Data Mining and Visualization

DATA MINING AND POWER SYSTEM

Application of Power Data Mining Within and Between Grid Fields

An Application of Data Mining in District Heating Substations for Improving Energy Performance

Application of Data Mining Method based on Deep Learning in Power System

An application of data mining in district heating substations f

Distributed Big Data Mining Platform for Smart Grid

Data mining technology and its application on power system

Power Big Data: New Assets of Electric Power Utilities

Data-driven Solutions for Power System Fault Analysis and Novelty Detection.

APPLICATION OF DATA WAREHOUSE AND DATA MINING TECHNIQUES TO POWER SYSTEMS

Data mining for energy systems: Review and prospect

Development of an Intelligent Coal Production and Operation Platform Based on a Real-Time Data Warehouse and AI Model

The Development and Application of Data Warehouse and Data Mining in Aluminum Electrolysis Control Systems