Abstract:Data representation plays a critical role in the performance of novelty detection (or ``anomaly detection'') methods in machine learning. The data representation of network traffic often determines the effectiveness of these models as much as the model itself. The wide range of novel events that network operators need to detect (e.g., attacks, malware, new applications, changes in traffic demands) introduces the possibility for a broad range of possible models and data representations. In each scenario, practitioners must spend significant effort extracting and engineering features that are most predictive for that situation or application. While anomaly detection is well-studied in computer networking, much existing work develops specific models that presume a particular representation -- often IPFIX/NetFlow. Yet, other representations may result in higher model accuracy, and the rise of programmable networks now makes it more practical to explore a broader range of representations. To facilitate such exploration, we develop a systematic framework, open-source toolkit, and public Python library that makes it both possible and easy to extract and generate features from network traffic and perform and end-to-end evaluation of these representations across most prevalent modern novelty detection models. We first develop and publicly release an open-source tool, an accompanying Python library (NetML), and end-to-end pipeline for novelty detection in network traffic. Second, we apply this tool to five different novelty detection problems in networking, across a range of scenarios from attack detection to novel device detection. Our findings general insights and guidelines concerning which features appear to be more appropriate for particular situations.

FastFE

FENXI: Deep-learning Traffic Analytics at the Edge

ServeFlow: A Fast-Slow Model Architecture for Network Traffic Analysis

Fast and Robust Online Traffic Classification Supporting Unseen Applications

Fast Packet Inspection Using State-Based Bloom Filter Engine

Feacan: Front-End Acceleration For Content-Aware Network Processing

An Input-Agnostic Hierarchical Deep Learning Framework for Traffic Fingerprinting

FASTeller: A Hardware Partial Aggregator for Accurate Flow Counting in Cloud Networks

Realtime Capture of High-Speed Traffic on Multi-Core Platform

Ceff: An Efficient Approach For Traffic Anomaly Detection And Classification

Tracking Millions of Flows in High Speed Networks for Application Identification

Feature Extraction for Novelty Detection in Network Traffic

FASTFLOW: Flexible Adaptive Congestion Control for High-Performance Datacenters

An early stage convolutional feature extracting method using for mining traffic detection

A Model of Encrypted Network Traffic Classification that Trades Off Accuracy and Efficiency

TEFE: A Time-Efficient Approach to Feature Extraction

FastFlow: Efficient Scalable Model-Driven Framework for Processing Massive Mobile Stream Data.

Fpga Based Low-Latency Market Data Feed Handler

HyperClassifier: Accurate, Extensible and Scalable Traffic Classification with Programmable Switches.

A Fast Deep Learning Method for Network Intrusion Detection Without Manual Feature Extraction