Abstract:In recent years, with the rapid development of sensing technology and the Internet of Things (IoT), sensors play increasingly important roles in traffic control, medical monitoring, industrial production and etc. They generated high volume of data in a streaming way that often need to be processed in real time. Therefore, streaming data computing technology plays an indispensable role in the real-time processing of sensor data in high throughput but low latency. However, there are two problems in deploying streaming data process ability in cloud computing data centre. Firstly, massive sensor nodes simultaneously upload data to the remote cloud computing data centre, which requires a large number of bandwidth resources supports. The existing network infrastructure cannot provide enough bandwidth at a reasonable price. Secondly, due to the geographical distribution characteristics of the cloud computing data centre, there will inevitably be large transmission delay during the process of data transmission. Such end-to-end delay is intolerable to mobile applications especially for those latency sensitive tasks. In view of the above problems, this paper proposes an autonomous driving oriented edge streaming data processing framework, which migrates the computing and storage capability from the remote cloud data centre to the edge data centre. It focuses on the change of vehicle flow in a specific geographical area, and uses the computing power sunk to edge node to process the massive streaming data generated by autonomous vehicles nearby. The proposed framework is implemented on top of Spark Streaming, which builds up a gray model based traffic flow monitor, a traffic prediction orientated prediction layer and a fuzzy control based Batch Interval dynamic adjustment layer for Spark Streaming. It could forecast the variation of sensors data arrive rate, make streaming Batch Interval adjustment in advance and implement real-time streaming process by edge. Therefore, it can realise the monitor and prediction of the data flow changes of the autonomous driving vehicle sensor data in geographical coverage of edge computing node area, meanwhile minimise the end-to-end latency but satisfy the application throughput requirements. The experiments show that it can predict short-term traffic with no more than 4% relative error in a whole day. By making batch consuming rate close to data generating rate, it can maintain system stability well even when arrival data rate changes rapidly. The Batch Interval can be converged to a suitable value in two minutes when data arrival rate is doubled. Compared with vanilla version Spark Streaming, where there has serious task accumulation and introduces large delay, it can reduce 35% latency by squeezing Batch Interval when data arrival rate is low; it also can significantly improve system throughput by only at most 25% Batch Interval increase when data arrival rate is high.

DS-Harmonizer: A Harmonization Service on Spatiotemporal Data Stream in Edge Computing Environment

Hadoop-based Framework for Big Data Analysis of Synchronised Harmonics in Active Distribution Network

Large-Scale Real-Time Semantic Processing Framework for Internet of Things

Elastic Streaming Semantic Engine for Web of Things.

AI-enabled IoT-Edge Data Analytics for Connected Living

Service Intelligence Oriented Distributed Data Stream Integration

Fast Wireless Sensor Anomaly Detection based on Data Stream in Edge Computing Enabled Smart Greenhouse

An Intelligent Edge Dual-Structure Ensemble Method for Data Stream Detection and Releasing

Edge Cloud Collaborative Stream Computing for Real-Time Structural Health Monitoring

Edge-Stream: a Stream Processing Approach for Distributed Applications on a Hierarchical Edge-computing System

Accelerating Remote Sensing Data Analysis Workflows by On-demand Processing

SymED: Adaptive and Online Symbolic Representation of Data on the Edge

A Real-Time Detection Method for Abnormal Data of Internet of Things Sensors Based on Mobile Edge Computing

A scalable software package for time series reconstruction of remote sensing datasets on the Google Earth Engine platform

3D Harmonic Loss: Towards Task-consistent and Time-friendly 3D Object Detection on Edge for V2X Orchestration

A Data Streaming Process Framework for Autonomous Driving By Edge

An edge streaming data processing framework for autonomous driving

The Streetscape Application Services Stack (SASS): Towards a Distributed Sensing Architecture for Urban Applications

The design of a streaming analytical workflow for processing massive transit feeds

Assembler: Efficient Discovery of Spatial Co-evolving Patterns in Massive Geo-sensory Data

Stream-DBSCAN: A Streaming Distributed Clustering Model for Water Quality Monitoring