Abstract:distributed weighted random sampling algorithm in distributed stream systems is novel.Multi-level query processing is presented to solve complex tasks and queries in distributed system.A synthetic processing topology strategy is provided to merge streams on partial repeated computing of overlap data.Early results with statistical estimations are continuously sent to users. Interactive query processing aims at generating approximate results with minimum response time. However, it is quite difficult for a batch-oriented processing system to progressively provide cumulatively accurate results in the context of a distributed environment. MapReduce Online extends the MapReduce framework to support online aggregation, but it is hindered by its processing speed in keeping up with ongoing real-time data events. We deploy the online aggregation algorithm over S4, a scalable stream processing system that is inspired by the combined functionalities of MapReduce and Actor model. Our system applies an asynchronous message communication mechanism from actor model to support online aggregation. It can process large scale data stream with high concurrency in a short response time. In this system, we adopt a distributed weighted random sampling algorithm to solve biased distribution between different streams. Furthermore, a multi-level query processing topology is developed to reduce overlapped processing for multiple queries. Our system can provide continuous window aggregation with a confidence interval and error bound. We have implemented our system and conducted plentiful experiments over the TPC-H benchmark. A large number of experiments are carried out to demonstrate that by using our system, high-quality query results can be generated within a short response time and that the approach outperforms MapReduce Online on data streams.

Load Shedding for Shared Window Joins over Data Streams

Load Shedding in Stream Databases: A Control-Based Approach.

Using Control Theory To Guide Load Shedding In Medical Data Stream Management System

Data Based Application Partitioning and Workload Balance in Distributed Environment

Progressive online aggregation in a distributed stream system

Diversity-Based Load Shedding Strategy over Pattern Streams

Using Control Theory for Load Shedding in Data Stream Management

A state lossless scheduling strategy in distributed stream computing systems

Multi-Way Windowed Streams Θ-Joins Using Cluster

An Efficient Scheduling Algorithm for Stream Computing.

An On-the-Fly Scheduling Strategy for Distributed Stream Processing Platform.

Parallelizing stateful operators in a distributed stream processing system: how, should you and how much?

PMJoin: Optimizing Distributed Multi-way Stream Joins by Stream Partitioning

An Integrated DBP for Streams with ( M, K )-Firm Real-Time Guarantee

Distributed scheduling and storage scheme based on LSM-OCTree for spatiotemporal stream

LOCI: Load Shedding Through Class-Preserving Data Acquisition.

An Efficient Approach of Processing Multiple Continuous Queries

Lc‐Stream: An elastic scheduling strategy with latency constraints in geo‐distributed stream computing environments

Maintaining Aggregate Views over Streams

Quality-driven processing of sliding window aggregates over out-of-order data streams

Adaptive Processing Algorithm of Ad Hoc Query in Data Streams