Abstract:Real-time surveillance systems, telecommunication systems, and other dynamic environments often generate tremendous (potentially infinite) volume of stream data: the volume is too huge to be scanned multiple times. Much of such data resides at rather low level of abstraction, whereas most analysts are interested in relatively high-level dynamic changes (such as trends and outliers). To discover such high-level characteristics, one may need to perform on-line multi-level, multi-dimensional analytical processing of stream data. In this paper, we propose an architecture, called stream_cube, to facilitate on-line, multi-dimensional, multi-level analysis of stream data. For fast online multi-dimensional analysis of stream data, three important techniques are proposed for efficient and effective computation of stream cubes. First, a tilted time frame model is proposed as a multi-resolution model to register time-related data: the more recent data are registered at finer resolution, whereas the more distant data are registered at coarser resolution. This design reduces the overall storage of time-related data and adapts nicely to the data analysis tasks commonly encountered in practice. Second, instead of materializing cuboids at all levels, we propose to maintain a small number of critical layers. Flexible analysis can be efficiently performed based on the concept of observation layer and minimal interesting layer . Third, an efficient stream data cubing algorithm is developed which computes only the layers (cuboids) along a popular path and leaves the other cuboids for query-driven, on-line computation. Based on this design methodology, stream data cube can be constructed and maintained incrementally with a reasonable amount of memory, computation cost, and query response time. This is verified by our substantial performance study.

DHS: Adaptive Memory Layout Organization of Sketch Slots for Fast and Accurate Data Stream Processing

An Efficient Sampling And Classification Approach For Flow Detection In Sdn-Based Big Data Centers

Diamond Sketch: Accurate Per-flow Measurement for Big Streaming Data

Hierarchical Sketch: an Efficient Solution for Threshold-t Flows Measurement in High-Speed Networks

Energy-efficient Pipelined DTW Architecture on Hybrid Embedded Platforms

HeavyCache: A Generic Sketch for Summarizing Data Streams

An Accurate Estimation Algorithm for Big Data Streams.

Adaptive Scheduling Strategy For Data Stream Management System

The Stair Sketch: Bringing More Clarity to Memorize Recent Events

HistSketch: A Compact Data Structure for Accurate Per-Key Distribution Monitoring.

A hybrid local and distributed sketching design for accurate and scalable heavy key detection in network data streams

Couper: Memory-Efficient Cardinality Estimation under Unbalanced Distribution.

Stream Cube: an Architecture for Multi-Dimensional Analysis of Data Streams

Hids: A Multifunctional Generator Of Hierarchical Data Streams

Enhancing Accuracy for Super Spreader Identification in High-Speed Data Streams

Finding frequent items in data streams using hierarchical information.

In Search of a Memory-Efficient Framework for Online Cardinality Estimation

Dynamic Sketch: Efficient and Adjustable Heavy Hitter Detection for Software Packet Processing

HybridSketch: A Memory-centric Precise Approach for Flow Measurement

An Adaptive Density Data Stream Clustering Algorithm