Hayden Jananthan,Jeremy Kepner,Michael Jones,William Arcand,David Bestor,William Bergeron,Chansup Byun,Timothy Davis,Vijay Gadepally,Daniel Grant,Michael Houle,Matthew Hubbell,Anna Klein,Lauren Milechin,Guillermo Morales,Andrew Morris,Julie Mullen,Ritesh Patel,Alex Pentland,Sandeep Pisharody,Andrew Prout,Albert Reuther,Antonio Rosa,Siddharth Samsi,Tyler Trigg,Gabriel Wachman,Charles Yee,Peter Michaleas

Abstract:Expanding the scientific tools available to protect computer networks can be aided by a deeper understanding of the underlying statistical distributions of network traffic and their potential geometric interpretations. Analyses of large scale network observations provide a unique window into studying those underlying statistics. Newly developed GraphBLAS hypersparse matrices and D4M associative array technologies enable the efficient anonymized analysis of network traffic on the scale of trillions of events. This work analyzes over 100,000,000,000 anonymized packets from the largest Internet telescope (CAIDA) and over 10,000,000 anonymized sources from the largest commercial honeyfarm (GreyNoise). Neither CAIDA nor GreyNoise actively emit Internet traffic and provide distinct observations of unsolicited Internet traffic (primarily botnets and scanners). Analysis of these observations confirms the previously observed Cauchy-like distributions describing temporal correlations between Internet sources. The Gull lighthouse problem is a well-known geometric characterization of the standard Cauchy distribution and motivates a potential geometric interpretation for Internet observations. This work generalizes the Gull lighthouse problem to accommodate larger classes of coastlines, deriving a closed-form solution for the resulting probability distributions, stating and examining the inverse problem of identifying an appropriate coastline given a continuous probability distribution, identifying a geometric heuristic for solving this problem computationally, and applying that heuristic to examine the temporal geometry of different subsets of network observations. Application of this method to the CAIDA and GreyNoise data reveals a several orders of magnitude difference between known benign and other traffic which can lead to potentially novel ways to protect networks.

Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays

Hypersparse Traffic Matrix Construction using GraphBLAS on a DPU

GraphBLAS on the Edge: Anonymized High Performance Streaming of Network Traffic

Hypersparse Network Flow Analysis of Packets with GraphBLAS

Focusing and Calibration of Large Scale Network Sensors using GraphBLAS Anonymized Hypersparse Matrices

Hyperscaling Internet Graph Analysis with D4M on the MIT SuperCloud

Hypersparse Traffic Matrices from Suricata Network Flows using GraphBLAS

75,000,000,000 Streaming Inserts/Second Using Hierarchical Hypersparse GraphBLAS Matrices

Vertical, Temporal, and Horizontal Scaling of Hierarchical Hypersparse GraphBLAS Matrices

Enabling Massive Deep Neural Networks with the GraphBLAS

GraphMat: High performance graph analytics made productive

Anonymized Network Sensing Graph Challenge

Mapping of Internet "Coastlines" via Large Scale Anonymized Network Source Correlations

Evaluation and Analysis of Distributed Graph-Parallel Processing Frameworks

A Billion Updates per Second Using 30,000 Hierarchical In-Memory D4M Databases

AMON: An Open Source Architecture for Online Monitoring, Statistical Analysis and Forensics of Multi-gigabit Streams

Analytic Queries over Geospatial Time-Series Data Using Distributed Hash Tables

Real-Time And Distributed Av Content Analysis System For Consumer Electronics Networks

Design and Implementation for Nonblocking Execution in GraphBLAS: Tradeoffs and Performance

Development of Web-Based Analysis System for Real-Time Decision Support on Arterial Networks

Graphulo: Linear Algebra Graph Kernels for NoSQL Databases