Abstract:As deep learning, virtual reality, and other technologies become mature, real-time data processing applications running on intelligent terminals are emerging endlessly; meanwhile, edge computing has developed rapidly and has become a popular research direction in the field of distributed computing. Edge computing network is a network computing environment composed of multi-edge computing nodes and data centers. First, the edge computing framework and key technologies are analyzed to improve the performance of real-time data processing applications. In the system scenario where the collaborative deployment tasks of multi-edge nodes and data centers are considered, the stream processing task deployment process is formally described, and an efficient multi-edge node-computing center collaborative task deployment algorithm is proposed, which solves the problem of copy-free task deployment in the task deployment problem. Furthermore, a heterogeneous edge collaborative storage mechanism with tight coupling of computing and data is proposed, which solves the contradiction between the limited computing and storage capabilities of data and intelligent terminals, thereby improving the performance of data processing applications. Here, a Feasible Solution (FS) algorithm is designed to solve the problem of placing copy-free data processing tasks in the system. The FS algorithm has excellent results once considering the overall coordination. Under light load, the V value is reduced by 73% compared to the Only Data Center-available (ODC) algorithm and 41% compared to the Hash algorithm. Under heavy load, the V value is reduced by 66% compared to the ODC algorithm and 35% compared to the Hash algorithm. The algorithm has achieved good results after considering the overall coordination and cooperation and can more effectively use the bandwidth of edge nodes to transmit and process data stream, so that more tasks can be deployed in edge computing nodes, thereby saving time for data transmission to the data centers. The end-to-end collaborative real-time data processing task scheduling mechanism proposed here can effectively avoid the disadvantages of long waiting times and unable to obtain the required data, which significantly improves the success rate of the task and thus ensures the performance of real-time data processing.

Real-time data stream processing and key techniques oriented to large-scale sensor data

Real-Time Big Data Processing Framework: Challenges and Solutions

Distributed storage and processing of real-time data in smart grid dispatching and control systems

Large-Scale Real-Time Semantic Processing Framework for Internet of Things

MapReduce for Large-Scale Monitor Data Analyses

Real-time Intelligent Big Data Processing:Technology, Platform, and Applications

Real-time intelligent big data processing: technology, platform, and applications

Progressive online aggregation in a distributed stream system

Large-scale Real-time Data-driven Scientific Applications

A task-level adaptive MapReduce framework for real-time streaming data in healthcare applications.

A Real-Time Scheduling Strategy Based on Processing Framework of Hadoop

Research of a MapReduce Model to Process the Traffic Big Data

The performance of MapReduce: an in-depth study

The Performance of MapReduce

Distributed data management using MapReduce

R-Store: A scalable distributed system for supporting real-time analytics

Evaluating Large Graph Processing in MapReduce Based on Message Passing

Research and Implementation of Multi-Terabyte Data Real-Time Loading Technology

A review on big data real-time stream processing and its scheduling techniques

Big Data Real-Time Processing Based on Storm

AI-enabled IoT-Edge Data Analytics for Connected Living