Abstract:SpeedStream is a universal distributed platform that can handle with massive data flows with the features of low coupling, high availability, low latency and high scalability. Focusing on the core technologies of real-time stream computing platform in cloud environment, this paper conducts a series of researches and implementation of the system. First of all, aiming at the availability of real-time streaming computing platform, we design a high availability framework based on Zookeeper. It ensures fault detection and recovery of process level and node level timely by monitoring heartbreak of each modules and strategy of fault migration. Secondly, in order to increase the application types of the platform, by means of directed cycle detection and iteration protection, we design a real-time streaming computing model that based on directed graph with sources and sinks, which can not only satisfy the needs of common DAG computing services, but also support iteration computing services including directed cycle, bidirectional arcs and annular arcs. In addition, the platform can realize personalized task scheduling strategy for users by establishing task allocation matrix and optimize task allocation model. Finally, in order to solve the many-to-many dynamic load-balancing between tasks, we apply scheduler with status and distributed session table. It overcomes the difficulty of maintaining consistency of session without global session table. We also testified the convergence of this method. The experiment indicates that the throughput and data processing delay of SpeedStream are superior to other alternatives in dealing with the businesses of iteration applications, high traffic fluctuation applications, and high demand of load-balancing applications. This platform provides reliable, universal, and real-time solutions to process massive data flows, such as to process the real-time trading data in e-commerce, to analyze sensing flow in internet of things, and monitor traffics of the Internet.

Hardware-Conscious Stream Processing

TATA: Throughput-Aware TAsk Placement in Heterogeneous Stream Processing with Deep Reinforcement Learning

Analyzing efficient stream processing on modern hardware

Video Stream Processing on a High Performance Reconfigurable Architecture

Energy-efficient Pipelined DTW Architecture on Hybrid Embedded Platforms

A holistic approach to build real-time stream processing system with GPU

Atomic Streaming: A Framework of On-Chip Data Supply System for Task-Parallel Mpsocs

Exploiting Stable Data Dependency in Stream Processing Acceleration on FPGAs

A Hardware/Software Method for Heterogeneous Cores Cooperating on Stream Architecture

StreamPoP: Stream programming oriented power-efficient audio DSP

Data-Aware Adaptive Compression for Stream Processing

Data Stream Processing for Packet-Level Analytics

Fine-Grained Multi-Query Stream Processing on Integrated Architectures

A Fast Hybrid Cache System for Pipelined Stream Processing

A catalog of stream processing optimizations

A Hardware Co-design Workflow for Scientific Instruments at the Edge

SpeedStream: A real-time stream data processing platform in the cloud.

Parallel Work System for Image Processing Based on Stream Processor and Its Algorithm of Stream-Oriented

CStream: Parallel Data Stream Compression on Multicore Edge Devices

Towards High-Performance Graph Processing: From a Hardware/Software Co-Design Perspective

Benchmarking Distributed Stream Data Processing Systems