Abstract:Today, smart grids, smart homes, smart water networks, and intelligent transportation, are infrastructure systems that connect our world more than we ever thought possible and are associated with a single concept, the Internet of Things (IoT). The number of devices connected to the IoT and hence the number of traffic flow increases continuously, as well as the emergence of new applications. Although cutting-edge hardware technology can be employed to achieve a fast implementation to handle this huge data streams, there will always be a limit on size of traffic supported by a given architecture. However, recent cloudbased big data technologies fortunately offer an ideal environment to handle this issue. Moreover, the ever-increasing high volume of traffic created on demand presents great challenges for flow management. As a solution, flow aggregation decreases the number of flows needed to be processed by the network. The previous works in the literature prove that most of aggregation strategies designed for smart grids aim at optimizing system operation performance. They consider a common identifier to aggregate traffic on each device, having its independent static aggregation policy. In this paper, we propose a dynamic approach to aggregate flows based on traffic characteristics and device preferences. Our algorithm runs on a big data platform to provide an end-to-end network visibility of flows, which performs high-speed and high-volume computations to identify the clusters of similar flows and aggregate massive number of mice flows into a few meta-flows. Compared with existing solutions, our approach dynamically aggregates large number of such small flows into fewer flows, based on traffic characteristics and access node preferences. Using this approach, we alleviate the problem of processing a large amount of micro flows, and also significantly improve the accuracy of meeting the access node QoS demands. We conducted experiments, using a dataset of up to 100,000 flows, and studied the performance of our algorithm analytically. The experimental results are presented to show the promising effectiveness and scalability of our proposed approach.

Hypersparse Network Flow Analysis of Packets with GraphBLAS

GraphBLAS on the Edge: Anonymized High Performance Streaming of Network Traffic

Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays

Focusing and Calibration of Large Scale Network Sensors using GraphBLAS Anonymized Hypersparse Matrices

Hypersparse Traffic Matrices from Suricata Network Flows using GraphBLAS

Hypersparse Traffic Matrix Construction using GraphBLAS on a DPU

Mapping of Internet "Coastlines" via Large Scale Anonymized Network Source Correlations

Exploring Network-Wide Flow Data with Flowyager

Superflows: A New Tool for Forensic Network Flow Analysis

Hyperscaling Internet Graph Analysis with D4M on the MIT SuperCloud

75,000,000,000 Streaming Inserts/Second Using Hierarchical Hypersparse GraphBLAS Matrices

XFlow: Benchmarking Flow Behaviors over Graphs

Anonymized Network Sensing Graph Challenge

High Speed Traffic Archiving System For Flow Granularity Storage And Querying

Leveraging Spatial and Temporal Correlations for Network Traffic Compression

CFlow: A Learning-Based Compressive Flow Statistics Collection Scheme for SDNs.

Big Data Based Dynamic Flow Aggregation over 5G Network Slicing.

Scalable Network Traffic Visualization Using Compressed Graphs.

Bigflow: A General Optimization Layer for Distributed Computing Frameworks

Scalable Bayesian modeling, monitoring and analysis of dynamic network flow data

Continuous Flow Measurement with SuperFlow.