Abstract:An increasing number of big data applications in various domains generate datasets continuously, which must be processed for various purposes in a timely manner. As one of the most popular streaming data processing systems, Spark Streaming applies a batch-based mechanism, which receives real-time input data streams and divides the data into multiple batches before passing them to Spark processing engine. As such, inappropriate system configurations including batch interval and executor count may lead to unstable states, hence undermining the capability and efficiency of real-time computing. Hence, determining suitable configurations is crucial to the performance of such systems. Many machine learning- and search-based algorithms have been proposed to provide configuration recommendations for streaming applications where input data streams are fed at a constant speed, which, however, is extremely rare in practice. Most real-life streaming applications process data streams arriving at a time-varying rate and hence require real-time system monitoring and continuous configuration adjustment, which still remains largely unexplored. We propose a novel streaming optimization scheme based on Simultaneous Perturbation Stochastic Approximation (SPSA), referred to as NoStop, which dynamically tunes system configurations to optimize real-time system performance with negligible overhead and proved convergence. The performance superiority of NoStop is illustrated by real-life experiments in comparison with Bayesian Optimization and Spark Back Pressure solutions. Extensive experimental results show that NoStop is able to keep track of the changing pattern of input data in real time and provide optimal configuration settings to achieve the best system performance. This optimization scheme could also be applied to other streaming data processing engines with tunable parameters.

Performance-sensitive Components Exploration in Spark Streaming

Modeling and Simulation of Spark Streaming

On combining system and machine learning performance tuning for distributed data stream applications

A catalog of stream processing optimizations

NoStop: A Novel Configuration Optimization Scheme for Spark Streaming

Benchmarking Distributed Stream Data Processing Systems

SP-Partitioner: A novel partition method to handle intermediate data skew in spark streaming

Fixed-point Iteration Approach to Spark Scalable Performance Modeling and Evaluation

Apache Spark Streaming, Kafka and HarmonicIO: A Performance Benchmark and Architecture Comparison for Enterprise and Scientific Computing

Adaptive Scheduling Framework of Streaming Applications based on Resource Demand Prediction with Hybrid Algorithms

OPTIMIZATION FOR SPARK MISSION PERFORMANCE BASED ON DATA CHARACTERISTICS

Progressive online aggregation in a distributed stream system

Scalable real-time processing with Spark Streaming: implementation and design of a Car Information System

Parallelizing stateful operators in a distributed stream processing system: how, should you and how much?

A Survey on Spark Ecosystem for Big Data Processing

Analyzing efficient stream processing on modern hardware

A Comprehensive Benchmarking Analysis of Fault Recovery in Stream Processing Frameworks

Exploring Real-Time Data Processing Using Big Data Frameworks

Model-driven Scheduling for Distributed Stream Processing Systems

Design and Implementation of Real Time Data Processing System Based on Spark Streaming

Data Stream Processing for Packet-Level Analytics