Abstract:In the coming big data era, stress test to IT systems under extreme data volume is crucial to the adoption of computing technologies in every corner of the cyber world. Appropriately generated benchmark datasets provide the possibility for administrators to evaluate the capacity of the systems when real datasets hard obtained have not extreme cases. Traditional benchmark data generators, however, mainly target at producing relation tables of arbitrary size following fixed distributions. The output of such generators are insufficient when it is used to measure the stability of the architecture with extremely dynamic and heavy workloads, caused by complicated/hiden factors in the generation mechanism of real world, e.g. dependency between stocks in the trading market and collaborative human behaviors on the social network. In this paper, we present a new framework, called Chronos, to support new demands on streaming data benchmarking, by generating and simulating realistic and fast data streams in an elastic manner. Given a small group of samples with timestamps, Chronos reproduces new data streams with similar characteristics of the samples, preserving column-wise correlations, temporal dependency and order statistics of the snapshot distributions at the same time. To achieve such realistic requirements, we propose 1) a column decomposition optimization technique to partition the original relation table into small sub-tables with minimal correlation information loss, 2) a generative and extensible model based on Latent Dirichlet Allocation to capture temporal dependency while preserving order statistics of the snapshot distribution, and 3) a new generation and assembling method to efficiently build tuples following the expected distribution on the snapshots. To fulfill the vision of elasticity, we also present a new parallel stream data generation mechanism, facilitating distributed nodes to collaboratively generate tuples with minimal synchronization overhead and excellent load balancing. Our extensive experimental studies on real world data domains confirm the efficiency and effectiveness of Chronos on stream benchmark generation and simulation.

MicroStream: A Distributed In-memory Caching Service for Data Production

TSCache

DartCache:a HashMap-Based Distributed Cache

Cache Optimizations of Distributed Storage for Software Streaming Services

SpeedStream: A real-time stream data processing platform in the cloud.

DASH: A duplication-aware flash cache architecture in virtualization environment

Cache streamization for high performance stream processor

Advanced Strategies for Precise and Transparent Debugging of Performance Issues in In-Memory Data Store-Based Microservices

Progressive online aggregation in a distributed stream system

E3: an Elastic Execution Engine for Scalable Data Processing.

Architectural Impact on Performance of In-memory Data Analytics: Apache Spark Case Study

Data-Aware Adaptive Compression for Stream Processing

Design and Implementation of the MaxStream Federated Stream Processing Architecture

Stream-Based Data Placement for Near-Data Processing with Extended Memory

Chronos: an Elastic Parallel Framework for Stream Benchmark Generation and Simulation.

In-memory big data analytics under space constraints using dynamic programming.

HeavyCache: A Generic Sketch for Summarizing Data Streams

LMStream: When Distributed Micro-Batch Stream Processing Systems Meet GPU

SSC: Cloud-Based Data Stream Management in Distributed Environments

Stream Cube: an Architecture for Multi-Dimensional Analysis of Data Streams

A Data-Driven Architecture Design of Stream Computing for the Dispatch and Control System of the Power Grid