Abstract:In recent years, with the rapid development of sensing technology and the Internet of Things (IoT), sensors play increasingly important roles in traffic control, medical monitoring, industrial production and etc. They generated high volume of data in a streaming way that often need to be processed in real time. Therefore, streaming data computing technology plays an indispensable role in the real-time processing of sensor data in high throughput but low latency. In view of the above problems, the proposed framework is implemented on top of Spark Streaming, which builds up a gray model based traffic flow monitor, a traffic prediction orientated prediction layer and a fuzzy control based Batch Interval dynamic adjustment layer for Spark Streaming. It could forecast the variation of sensors data arrive rate, make streaming Batch Interval adjustment in advance and implement real-time streaming process by edge. Therefore, it can realize the monitor and prediction of the data flow changes of the autonomous driving vehicle sensor data in geographical coverage of edge computing node area, meanwhile minimize the end-to-end latency but satisfy the application throughput requirements. The experiments show that it can predict short-term traffic with no more than 4% relative error in a whole day. By making batch consuming rate close to data generating rate, it can maintain system stability well even when arrival data rate changes rapidly. The Batch Interval can be converged to a suitable value in two minutes when data arrival rate is doubled. Compared with vanilla version Spark Streaming, where there has serious task accumulation and introduces large delay, it can reduce 35% latency by squeezing Batch Interval when data arrival rate is low; it also can significantly improve system throughput by only at most 25% Batch Interval increase when data arrival rate is high.

Fast and Fine-grained Autoscaler for Streaming Jobs with Reinforcement Learning.

TATA: Throughput-Aware TAsk Placement in Heterogeneous Stream Processing with Deep Reinforcement Learning

Edge-Cloud Collaborative Streaming Video Analytics with Multi-agent Deep Reinforcement Learning

Generalizable Resource Allocation in Stream Processing via Deep Reinforcement Learning

Bayesian-Driven Automated Scaling in Stream Computing With Multiple QoS Targets

Adaptive Scheduling Framework of Streaming Applications based on Resource Demand Prediction with Hybrid Algorithms

Auto-tuning Distributed Stream Processing Systems using Reinforcement Learning

Deep-Reinforcement-Learning-based User-Preference-Aware Rate Adaptation for Video Streaming

DeepScaler: Holistic Autoscaling for Microservices Based on Spatiotemporal GNN with Adaptive Graph Learning

A Meta Reinforcement Learning Approach for Predictive Autoscaling in the Cloud

A Predictive Autoscaler for Elastic Batch Jobs

A Unified Replay-based Continuous Learning Framework for Spatio-Temporal Prediction on Streaming Data

Batch Adaptative Streaming for Video Analytics

Auto-Scaling Containerized Applications in Geo-Distributed Clouds

Intelligent Video Ingestion for Real-time Traffic Monitoring

Towards QoS-Aware Cloud Live Transcoding: A Deep Reinforcement Learning Approach

Scaleplus: Towards Fast Scaling of Distributed Streaming Dataflows.

MSARS: A Meta-Learning and Reinforcement Learning Framework for SLO Resource Allocation and Adaptive Scaling for Microservices

FairStream: Fair Multimedia Streaming Benchmark for Reinforcement Learning Agents

A Data Streaming Process Framework for Autonomous Driving By Edge

Fuzzy Allocation of Fine-Grained Compute Resources for Grid Data Streaming Applications.