Abstract:An increasing number of big data applications in various domains generate datasets continuously, which must be processed for various purposes in a timely manner. As one of the most popular streaming data processing systems, Spark Streaming applies a batch-based mechanism, which receives real-time input data streams and divides the data into multiple batches before passing them to Spark processing engine. As such, inappropriate system configurations including batch interval and executor count may lead to unstable states, hence undermining the capability and efficiency of real-time computing. Hence, determining suitable configurations is crucial to the performance of such systems. Many machine learning- and search-based algorithms have been proposed to provide configuration recommendations for streaming applications where input data streams are fed at a constant speed, which, however, is extremely rare in practice. Most real-life streaming applications process data streams arriving at a time-varying rate and hence require real-time system monitoring and continuous configuration adjustment, which still remains largely unexplored. We propose a novel streaming optimization scheme based on Simultaneous Perturbation Stochastic Approximation (SPSA), referred to as NoStop, which dynamically tunes system configurations to optimize real-time system performance with negligible overhead and proved convergence. The performance superiority of NoStop is illustrated by real-life experiments in comparison with Bayesian Optimization and Spark Back Pressure solutions. Extensive experimental results show that NoStop is able to keep track of the changing pattern of input data in real time and provide optimal configuration settings to achieve the best system performance. This optimization scheme could also be applied to other streaming data processing engines with tunable parameters.

Evaluating Trigger Conditions on Streaming Time Series with User-given Quality Requirements

Condition Evaluation for Speculative Systems: a Streaming Time Series Case.

Rate Optimization for Constant Quality Fine Granularity Scalable Video Streaming

Adaptive Video Streaming with Automatic Quality-of-Experience Optimization

Real-Time Scheduling Based On Imprecise Computation For Scalable Streaming Media System Over The Internet

Optimized Video Streaming over Cloud: A Stall-Quality Trade-off

A Time-Variant QoE Model Based on Real Video Streaming Data

High Efficiency Live Video Streaming With Frame Dropping

Rate Control Optimizing Model for Constraining Over-Saturated Live Streaming Quality

Imprecise Computation Scheduling On Scalable Media Stream Delivery

Energy-Efficient Traffic Rate Adaptation for Wireless Streaming Media Transmission

Delving into Internet Streaming Media Delivery

Quality-of-Experience Assessment for Ultra-low Latency Live Streaming Videos

Adaptive Scheduling Framework of Streaming Applications based on Resource Demand Prediction with Hybrid Algorithms

Analytical QoE Models for Bit-Rate Switching in Dynamic Adaptive Streaming Systems

Temporal Reasoning Guided QoE Evaluation for Mobile Live Video Broadcasting

A modeling and optimization framework for streaming multiple rate adaptive media

Quality-Aware Video Streaming for Green Cellular Networks With Hybrid Energy Sources

NoStop: A Novel Configuration Optimization Scheme for Spark Streaming

Towards Better Quality of Experience in HTTP Adaptive Streaming

Fresher Content or Smoother Playback? A Brownian-Approximation Framework for Scheduling Real-Time Wireless Video Streams