Abstract:In recent years, with the advancement of the national construction of smart grid and the development of power grid enterprise integration system, the traditional power data platform has some defect, such as insufficient scalability, repeated implementation of two sets of logic of off-line data warehouse and real-time data warehouse, highly correlated storage of data warehouse during computation, and difficulty in platform migration, which restrict the deeper mining and analysis of power big data. Therefore, this paper adopts a technical architecture incorporating distributed calculating, micro-service, flow-batch integration and lake-warehouse integration, combining the current popular Web front-end and back-end technologies such as Vue, SpringCloud and Flask, and utilizing big data components such as Hadoop, Flink, Hudi and Kafka as well as Docker containerization technology. We explore and build a power grid system operation oriented big data platform that assembles real-time computing, multi-source heterogeneous data storage, and develop APIs for authority management, information management, data analysis and visualization, machine learning and other data supporting services that assist power enterprises in managing electrical equipment and users' power consumption.With the help of this platform, power enterprises can intelligently track the equipment status, view the offline and real-time visual charts to analyze the power consumption by users, so as to have better basis in the power dispatching, maintenance and scheme adjustment of electrical market.

Research and Design of Real Time Big Data Visualization Analysis Platform Based on Flink

An efficient architecture for processing real-time traffic data streams using apache flink

Exploring Real-Time Data Processing Using Big Data Frameworks

A Practical Yet Accurate Real-Time Statistical Analysis Library for Hydrologic Time-Series Big Data

Socialanalysis: A Real-Time Query And Mining System From Social Media Data Streams

Design and Implementation of Real Time Data Processing System Based on Spark Streaming

Design and Implementation of Power Big Data Platform

Real-time intelligent big data processing: technology, platform, and applications

Real-time Intelligent Big Data Processing:Technology, Platform, and Applications

Real-time Event Joining in Practice With Kafka and Flink

Video2Flink: real-time video partitioning in Apache Flink and the cloud

Towards an Integrated Platform for Big Data Analysis

Real-Time Search Method for Large-Scale Regional Targets Based on Parallel Google S2 Algorithm.

Evaluating New Approaches of Big Data Analytics Frameworks

FlashView

Design and Implementation of Financial Big Data Visualization Analysis Platform

Technical Report: On the Usability of Hadoop MapReduce, Apache Spark & Apache Flink for Data Science

Analysis and Prediction of Big Stream Data in Real-Time Water Quality Monitoring System.

Real-time Processing Technology,platform and Application of Streaming Big Data

Traffic Data Processing At Age Of Big Data

GeoFlink: A Distributed and Scalable Framework for the Real-time Processing of Spatial Streams