Abstract:Recently proposed methods in intrusion detection are iterating on machine learning methods as a potential solution. These novel methods are validated on one or more datasets from a sparse collection of academic intrusion detection datasets. Their recognition as improvements to the state-of-the-art is largely dependent on whether they can demonstrate a reliable increase in classification metrics compared to similar works validated on the same datasets. Whether these increases are meaningful outside of the training/testing datasets is rarely asked and never investigated. This work aims to demonstrate that strong general performance does not typically follow from strong classification on the current intrusion detection datasets. Binary classification models from a range of algorithmic families are trained on the attack classes of CSE-CIC-IDS2018, a state-of-the-art intrusion detection dataset. After establishing baselines for each class at various points of data access, the same trained models are tasked with classifying samples from the corresponding attack classes in CIC-IDS2017, CIC-DoS2017 and CIC-DDoS2019. Contrary to what the baseline results would suggest, the models have rarely learned a generally applicable representation of their attack class. Stability and predictability of generalized model performance are central issues for all methods on all attack classes. Focusing only on the three best-in-class models in terms of interdataset generalization, reveals that for network-centric attack classes (brute force, denial of service and distributed denial of service), general representations can be learned with flat losses in classification performance (precision and recall) below 5%. Other attack classes vary in generalized performance from stark losses in recall (-35%) with intact precision (98+%) for botnets to total degradation of precision and moderate recall loss for Web attack and infiltration models. The core conclusion of this article is a warning to researchers in the field. Expecting results of proposed methods on the test sets of state-of-the-art intrusion detection datasets to translate to generalized performance is likely a serious overestimation. Four proposals to reduce this overestimation are set out as future work directions.

Exploring the effect of training-time randomness on the performance of deep neural networks for intrusion detection

An Intrusion Detection System Using a Deep Neural Network with Gated Recurrent Units.

Experimental Review of Neural-based approaches for Network Intrusion Management

Rolling the Dice for Better Deep Learning Performance: A Study of Randomness Techniques in Deep Neural Networks

Network Anomaly Intrusion Detection Based on Deep Learning Approach

A Real Time Deep Learning based Approach for Detecting Network Attacks

Intrusion Detection System with Machine Learning and Multiple Datasets

Machine learning to improve the performance of anomaly-based network intrusion detection in big data

IDS with deep learning techniques

A Performance Analysis Approach for Network Intrusion Detection Algorithms.

Attack vs Benign Network Intrusion Traffic Classification

Investigating Generalized Performance of Data-Constrained Supervised Machine Learning Models on Novel, Related Samples in Intrusion Detection

Research on Intrusion Detection Method Based on PGoogLeNet-IDS Model

A Deep Learning Approach for Intrusion Detection Using Recurrent Neural Networks

A Deep Random Forest Model on Spark for Network Intrusion Detection

LSTM Hyper-Parameter Selection for Malware Detection: Interaction Effects and Hierarchical Selection Approach

Empirical Enhancement of Intrusion Detection Systems: A Comprehensive Approach with Genetic Algorithm-based Hyperparameter Tuning and Hybrid Feature Selection

Addressing the class imbalance problem in network intrusion detection systems using data resampling and deep learning

RNNIDS: Enhancing Network Intrusion Detection Systems through Deep Learning

Evolving random weight neural networks based on oversampled-segmented examples for IoT intrusion detection