Abstract:ABSTRACTWe consider a distributed stream processing application, expressed as a data-flow graph with operators as vertices connected by streams and deployed over a cluster of compute nodes, where a small subset of the operators are often the performance bottlenecks for the entire application. In cases where a bottleneck operator is stateless, it is obvious that parallelization by splitting the incoming stream among multiple parallel operators deployed on different nodes can help improve performance. However, it is not so obvious when the bottleneck operator is stateful. In such a case, parallelization is much more challenging as it often requires a state sharing mechanism for the parallel operators. Moreover, it incurs additional overheads of required accesses by the parallel operators to shared state and synchronization constructs. In this paper, we propose a parallelization framework for stateful stream processing operators. The framework not only addresses issues related to the system model and support for operator parallelization, but also delves into the theoretical details that model the suitability of parallelization and the optimal degree of parallelism. We have implemented and evaluated our framework in the context of IBM's System S distributed stream processing middleware. While microbenchmarks are used to validate the proposed theoretical model, a parallelized implementation of a moving KNN application is used for the purpose of evaluation.

Parallel Work System for Image Processing Based on Stream Processor and Its Algorithm of Stream-Oriented

A parallel computing method for irregular work

Scalable-Grain Pipeline Parallelization Method For Multi-Core Systems

Simd Massively Parallel Processing System For Real-Time Image Processing

Novel many-core architecture design for real-time image processing

A Multi-Core Parallel Fusion Algorithm for Remote-Sensing Image

Video Stream Processing on a High Performance Reconfigurable Architecture

High-Performance Implementation of Stream Model Based H.264 Video Coding on Parallel Processors

Parallel Computing for Water Quality Equation Based on Jppf

Data Parallelism Exploiting for H.264 Encoder

A Hardware/Software Method for Heterogeneous Cores Cooperating on Stream Architecture

Parallelizing stateful operators in a distributed stream processing system: how, should you and how much?

A Mechanism of Remote Sensing Image for Parallel Processing Base on Splitting Blocks

Dsp-Based Parallel Implementation Of Speeded-Up Robust Features

A high speed multi-level-parallel array processor for vision chips

Parallel System Architecture of Multi-Dsp Interconnected by FPGA

Design and Experiment of Prototype System for Parallel Processing of Remote Sensing Images Based on FPGA

Multi-core Parallel Operation for Image Processing Acceleration

Design And Implementation Of Reconfigurable Stream Processor In Multimedia Applications

High-efficient software parallel CAVLC encoder based on programmable stream processor.

MASA Stream Architecture and Evaluating for a Fluid Computing Application