Abstract:Due to the massive amount of data in high-speed network traffic and the limit on processing capability, it is a great challenge to accurately measure and monitor network traffic over high-speed links online. A new data structure is presented in this paper for locating the hosts associated with large connection degrees or significant changes in connection degrees based on the reversible connection degree sketch to monitor anomalous network traffic. The reversible connection degree sketch builds a compact summary of host connection degrees efficiently and accurately. For each packet coming, it only needs to set several bits selected in a bit array by a group of hash functions. These hash functions are designed based on the Chinese Remainder Theorem so that the in-degree or out-degree associated with a given host can be accurately estimated. With this new data structure, we develop a new reverse sketch method for locating abnormal hosts. Although the reversible connection degree sketch does not preserve any host address information, we can analytically reconstruct the host addresses associated with large connection degrees or significant changes in connection degrees by a simple calculation purely based on the characteristics of the hash functions. Furthermore, a reinforced reversible connection degree sketch, the double connection degree sketch, is developed to reduce false positives which are commonly encountered in the sketch-based methods. A traffic monitoring system based on this double connection degree is developed to detect and classify the abnormal hosts associated with large connection degrees or significant changes in connection degrees. The experiments are conducted based on the actual network traffic and the testing results show that our method is accurate and efficient.

System and method for converting collected high-dimension data into low-dimension data

System and method for incrementally converting high dimensional data into low dimensional data

Method, apparatus and system for processing network data

Implementation of Heterogeneous Data Conversion and Information Distribution System

Data Compression and Storage under High Speed Network

IMPLEMENT DATA SHARING OVER NETWORK HETEROGENEOUS DATABASES BY DATA DIMENSION REDUCTION METHOD

A Method of Real-Time Data Acquisition and Exchange in Adaptation to Integrated Dispatching Automation System

Distributed storage and processing of real-time data in smart grid dispatching and control systems

Research of a New High-Performance Data-Collecting Transmissive System

Research On Storage And Retrieval Method Of Mass Data For High-Speed Train

Data processing method and device, computer equipment and storage medium

A Data Streaming Method for Monitoring Host Connection Degrees of High-Speed Links

A New Method for Handling Unstructured Data in the High-Speed Railway Passenger Service System

Parallel Spatial-Data Conversion Engine: Enabling Fast Sharing Of Massive Geospatial Data

A fast reading and processing method of batch human CT image data

A Fully Distributed Collection Technology for Mass Simulation Data

Research on Parallel Computing of Spatial Vector Data Conversion Based on Common Interface

A Distributed 2D-to-3d Video Conversion System

Real-time intelligent big data processing: technology, platform, and applications

Research on Diagnostic Information of Smart Medical Care Based on Big Data

Real-time Intelligent Big Data Processing:Technology, Platform, and Applications