Abstract:As deep learning, virtual reality, and other technologies become mature, real-time data processing applications running on intelligent terminals are emerging endlessly; meanwhile, edge computing has developed rapidly and has become a popular research direction in the field of distributed computing. Edge computing network is a network computing environment composed of multi-edge computing nodes and data centers. First, the edge computing framework and key technologies are analyzed to improve the performance of real-time data processing applications. In the system scenario where the collaborative deployment tasks of multi-edge nodes and data centers are considered, the stream processing task deployment process is formally described, and an efficient multi-edge node-computing center collaborative task deployment algorithm is proposed, which solves the problem of copy-free task deployment in the task deployment problem. Furthermore, a heterogeneous edge collaborative storage mechanism with tight coupling of computing and data is proposed, which solves the contradiction between the limited computing and storage capabilities of data and intelligent terminals, thereby improving the performance of data processing applications. Here, a Feasible Solution (FS) algorithm is designed to solve the problem of placing copy-free data processing tasks in the system. The FS algorithm has excellent results once considering the overall coordination. Under light load, the V value is reduced by 73% compared to the Only Data Center-available (ODC) algorithm and 41% compared to the Hash algorithm. Under heavy load, the V value is reduced by 66% compared to the ODC algorithm and 35% compared to the Hash algorithm. The algorithm has achieved good results after considering the overall coordination and cooperation and can more effectively use the bandwidth of edge nodes to transmit and process data stream, so that more tasks can be deployed in edge computing nodes, thereby saving time for data transmission to the data centers. The end-to-end collaborative real-time data processing task scheduling mechanism proposed here can effectively avoid the disadvantages of long waiting times and unable to obtain the required data, which significantly improves the success rate of the task and thus ensures the performance of real-time data processing.

Elastic high-performance computing platform for real-time data analysis

E3: an Elastic Execution Engine for Scalable Data Processing.

ECHO: An Adaptive Orchestration Platform for Hybrid Dataflows across Cloud and Edge

A Distributed Multi Agents Based Platform for High Performance Computing Infrastructures

Octopus: Experiences with a Hybrid Event-Driven Architecture for Distributed Scientific Computing

ElasticBroker: Combining HPC with Cloud to Provide Realtime Insights into Simulations

Exploring Real-Time Data Processing Using Big Data Frameworks

Large-Scale Real-Time Semantic Processing Framework for Internet of Things

Developing an edge computing platform for real-time descriptive analytics

Enabling Workload Engineering in Edge, Fog, and Cloud Computing through OpenStack-based Middleware

Design and implementation of a cloud-based event-driven architecture for real-time data processing in wireless sensor networks

SpeedStream: A real-time stream data processing platform in the cloud.

Hetero-Edge: Orchestration of Real-time Vision Applications on Heterogeneous Edge Clouds

Providing Scalable Database Services On The Cloud

An Event-Driven Approach to Serverless Seismic Imaging in the Cloud

A containerized analytics framework for data and compute-intensive pipeline applications

An enhanced distributed framework for real-time performance testing of large scale IoT dataset using big data analytic tools

The Stratosphere platform for big data analytics

AI-enabled IoT-Edge Data Analytics for Connected Living

Efficient, Dynamic Multi-tenant Edge Computation in EdgeOS

A reference architecture for serverless big data processing