Abstract:SpeedStream is a universal distributed platform that can handle with massive data flows with the features of low coupling, high availability, low latency and high scalability. Focusing on the core technologies of real-time stream computing platform in cloud environment, this paper conducts a series of researches and implementation of the system. First of all, aiming at the availability of real-time streaming computing platform, we design a high availability framework based on Zookeeper. It ensures fault detection and recovery of process level and node level timely by monitoring heartbreak of each modules and strategy of fault migration. Secondly, in order to increase the application types of the platform, by means of directed cycle detection and iteration protection, we design a real-time streaming computing model that based on directed graph with sources and sinks, which can not only satisfy the needs of common DAG computing services, but also support iteration computing services including directed cycle, bidirectional arcs and annular arcs. In addition, the platform can realize personalized task scheduling strategy for users by establishing task allocation matrix and optimize task allocation model. Finally, in order to solve the many-to-many dynamic load-balancing between tasks, we apply scheduler with status and distributed session table. It overcomes the difficulty of maintaining consistency of session without global session table. We also testified the convergence of this method. The experiment indicates that the throughput and data processing delay of SpeedStream are superior to other alternatives in dealing with the businesses of iteration applications, high traffic fluctuation applications, and high demand of load-balancing applications. This platform provides reliable, universal, and real-time solutions to process massive data flows, such as to process the real-time trading data in e-commerce, to analyze sensing flow in internet of things, and monitor traffics of the Internet.

A Task Scheduling Approach for Real-Time Stream Processing

TATA: Throughput-Aware TAsk Placement in Heterogeneous Stream Processing with Deep Reinforcement Learning

Communication-Efficient Task Scheduling for Real-Time Distributed Computing.

An On-the-Fly Scheduling Strategy for Distributed Stream Processing Platform.

New Scheduling Algorithm for Real-Time Multiprocessor Systems Aimed to Independent and Periodic Multi-Tasks

Application and Research of Dynamic Scheduling Algorithm for Multiprocessor

Allocation and Scheduling of Strictly Periodic Tasks in Multi-core Real-Time Systems

Topology-Aware Task Allocation For Distributed Stream Processing With Latency Guarantee

The Real-Time Scheduling Strategy Based on Traffic and Load Balancing in Storm

Stromax: Partitioning-Based Scheduler For Real-Time Stream Processing System

Runtime-Aware Adaptive Scheduling in Stream Processing

A Real-Time Scheduling Strategy Based on Processing Framework of Hadoop

Topology-aware task allocation for online distributed stream processing applications with latency constraints

The Research Of Real-Time Scheduling For Data Stream Operators

Two-stage Scheduling of Stream Computing for Industrial Cloud-edge Collaboration

An Efficient Scheduling Algorithm for Stream Computing.

An Integrated Dynamic Scheduling Algorithm with Scheduling Strategy in Real-time Heterogeneous Systems

Multi-Dimensional Scheduling for Real-Time Tasks on Heterogeneous Clusters.

Big Data Processing Workflows Oriented Real-Time Scheduling Algorithm using Task-Duplication in Geo-Distributed Clouds

Streaming Task Graph Scheduling for Dataflow Architectures

SpeedStream: A real-time stream data processing platform in the cloud.