Abstract:Unknown cyber-attacks have appeared constantly. Several anomaly detection techniques based on semi-supervised learning have been proposed to detect these unknown cyber-attacks. Among them, the Denoising Auto-Encoder (DAE) scheme performs better than others in accuracy but is not good enough in precision. This paper proposes a novel two-stage deep learning structure for network flow anomaly detection by combining the models of Gate Recurrent Unit (GRU) and DAE. By using supervised anomaly detection with a selection mechanism to assist semi-supervised anomaly detection, the precision and accuracy of the anomaly detection system are improved. In the proposed structure, we first use the GRU model to analyze the network flow and then take the outcome from the Softmax function as a confidence score. When the score is more than or equal to the predefined confidence threshold, the GRU model outputs the flow as a positive result, no matter the flow is classified as normal or abnormal. When the score is less than the confidence threshold, GRU model outputs the flow as a negative result and passes the flow to DAE model for flow classification. DAE then determines a reconstruction error threshold by learning the pattern of normal flows. Accordingly, the flow is normal or abnormal depending on whether it is under or over the reconstruction error threshold. A comparative experiment is performed using NSL-KDD dataset as benchmark. The results revealed that the precision using the proposed scheme is 0.83% better than DAE. The accuracy using the proposed approach is 90.21%, which is better than Random Forest, Naïve Bayes, One-Dimensional Convolutional Neural Network, two-stage Auto-Encoder, etc. In addition, the proposed approach is also applied to the environment of software defined network (SDN). By adopting our approach in SDN environment, the precision and F-measure are significantly improved.

Large-Scale Ip Network Flow-Level Abnormal Behavior Trends Analysis Using Combined Anomaly Grey Forecasting Algorithm

Apply Anomaly Grey Forecasting Algorithm to Cyberspace Situation Prediction

AGNP: Network-wide Short-Term Probabilistic Traffic Speed Prediction and Imputation

Large-scale IP Network Behavior Anomaly Detection and Identification Using Substructure-Based Approach and Multivariate Time Series Mining

Abnormal Network Traffic Analysis Based on IP Address Clustering

Anomaly detection application on IP backbone using time series forecast.

Multiple Time Series Data Mining Approach of Large-Scale IP Network Abnormal Traffic Feature Analysis

Monitoring Abnormal Network Traffic Based on Blind Source Separation Approach

Anomaly Detection for Power Grid Based on Network Flow.

A Transform Domain-Based Anomaly Detection Approach To Network-Wide Traffic

Detection Network Anomalies Based on Packet and Flow Analysis

Analysis Method of Multi-Source Flow Characteristics and Its Application in Anomaly Detection

Simulation Research of Wireless Network Data Flow Anomaly Detection

A Standardized ICS Network Data Processing Flow with Generative Model in Anomaly Detection

An empirical study for the traffic flow rate prediction-based anomaly detection in software-defined networking: a challenging overview

Network Anomaly Traffic Detection Based on IP Monitoring and Non-Gaussian Statistics

Network Anomaly Detection Based on Statistical Approach and Time Series Analysis

A Novel Two-Stage Deep Learning Structure for Network Flow Anomaly Detection

A novel spread estimation based abnormal flow detection in high-speed networks

Dynamic Group Behavior Analysis and Its Application in Network Abnormal Behavior Detection.

Monitoring abnormal traffic flows based on independent component analysis