Abstract:We describe and validate a novel data-driven approach to the real time detection and classification of traffic anomalies based on the identification of atypical fluctuations in the relationship between density and flow. For aggregated data under stationary conditions, flow and density are related by the fundamental diagram. However, high resolution data obtained from modern sensor networks is generally non-stationary and disaggregated. Such data consequently show significant statistical fluctuations. These fluctuations are best described using a bivariate probability distribution in the density-flow plane. By applying kernel density estimation to high-volume data from the UK National Traffic Information Service (NTIS), we empirically construct these distributions for London's M25 motorway. Curves in the density-flow plane are then constructed, analogous to quantiles of univariate distributions. These curves quantitatively separate atypical fluctuations from typical traffic states. Although the algorithm identifies anomalies in general rather than specific events, we find that fluctuations outside the 95\% probability curve correlate strongly with the spikes in travel time associated with significant congestion events. Moreover, the size of an excursion from the typical region provides a simple, real-time measure of the severity of detected anomalies. We validate the algorithm by benchmarking its ability to identify labelled events in historical NTIS data against some commonly used methods from the literature. Detection rate, time-to-detect and false alarm rate are used as metrics and found to be generally comparable except in situations when the speed distribution is bi-modal. In such situations, the new algorithm achieves a much lower false alarm rate without suffering significant degradation on the other metrics. This method has the additional advantage of being self-calibrating.

A Flow-Based Anomaly Detection Method Using Entropy and Multiple Traffic Features

Packet Payload Based Anomalous Network Intrusion Detection

Analysis Method of Multi-Source Flow Characteristics and Its Application in Anomaly Detection

Discovering Anomaly on the Basis of Flow Estimation of Alert Feature Distribution

Anomaly network traffic detection algorithm based on information entropy measurement under the cloud computing environment

High-speed anomaly traffic detection based on staged frequency domain features

Characterizing Network Anomaly Traffic with Euclidean Distance-Based Multiscale Fuzzy Entropy.

Adjustable Piecewise Entropy for Network Traffic Anomaly Detection

Monitoring Abnormal Network Traffic Based on Blind Source Separation Approach

An On-Line Anomaly Detection Method Based on a New Stationary Metric - Entropy-Ratio

A New Anomaly Detection Method Based on IGTE and IGFE.

Semi-Supervised Learning for Anomaly Traffic Detection via Bidirectional Normalizing Flows

FlowGANAnomaly: Flow-Based Anomaly Network Intrusion Detection with Adversarial Learning

Anomaly Detection Based on Traffic Information Structure

Graph-Augmented Normalizing Flows for Anomaly Detection of Multiple Time Series

Research on the detection of abnormal traffic for multi-channel network

A multi-channel anomaly detection method with feature selection and multi-scale analysis

Anomaly detection and classification in traffic flow data from fluctuations in the flow-density relationship

Flow-based intrusion detection on software-defined networks: a multivariate time series anomaly detection approach

Research on Dynamic Data Flow Anomaly Detection based on Machine Learning

Unsupervised Abnormal Traffic Detection through Topological Flow Analysis