Abstract:A key function for network traffic monitoring and analysis is the ability to perform aggregate queries over multiple data streams. Change detection is an important primitive which can be extended to construct many aggregate queries. The recently proposed sketches [1] are among the very few that can detect heavy changes online for high speed links, and thus support various aggregate queries in both temporal and spatial domains. However, it does not preserve the keys (e.g., source IP address) of flows, making it difficult to reconstruct the desired set of anomalous keys. In an earlier abstract we proposed a framework for a reversible sketch data structure that offers hope for efficient extraction of keys [2]. However, this scheme is only able to detect a single heavy change key and places restrictions on the statistical properties of the key space. To address these challenges, we propose an efficient reverse hashing scheme to infer the keys of culprit flows from reversible sketches. There are two phases. The first operates online, recording the packet stream in a compact representation with negligible extra memory and few extra memory accesses. Our prototype single FPGA board implementation can achieve a throughput of over 16 Gbps for 40-byte-packet streams (the worst case). The second phase identifies heavy changes and their keys from the representation in nearly real time. We evaluate our scheme using traces from large edge routers with OC-12 or higher links. Both the analytical and experimental results show that we are able to achieve online traffic monitoring and accurate change/intrusion detection over massive data streams on high speed links, all in a manner that scales to large key space size. To the best of our knowledge, our system is the first to achieve these properties simultaneously.

High Speed Traffic Archiving System For Flow Granularity Storage And Querying

TIFAflow: Enhancing Traffic Archiving System with Flow Granularity for Forensic Analysis in Network Security

A Survey on Internet Traffic Archival Systems

An Efficient Indexing Scheme for Network Traffic Collection and Retrieval System

Hardware-Based Algorithm Acceleration Towards Efficient Network Traffic Storage Systems

Reverse Hashing For High-Speed Network Monitoring: Algorithms, Evaluation, And Applications

ANTI: an Adaptive Network Traffic Indexing Algorithm for High-Speed Networks

A Real-Time Network Traffic Profiling System

SMART: a System for Online Monitoring Large Volumes of Network Traffic

Tracking Millions of Flows in High Speed Networks for Application Identification

TIFA: Enabling Real-Time Querying and Storage of Massive Stream Data

Network Traffic Capture and Storage for IPv6 Network

Novel traffic measurement architecture for high-speed IP network

CAMP: A New Bitmap Index for Data Retrieval in Traffic Archival.

An Architecture for Parallelizing Network Monitoring Based on Multi-Core Processors

Design and implementation of a high-speed network traffic measurement system

Monitoring Flow-level High-speed Data Streams with Reversible Sketches

Commodity PC Based on Flow Aggregation System for Gigabit Networks

A Method of Net Flow Index Retrieval and Compression based on Inverted List

Design and Implementation of Network Traffic Analysis System

Network Traffic Analysis System Based on IPFIX