Abstract:Variable selection (also known as feature selection ) is essential to optimize the learning complexity by prioritizing features, particularly for a massive, high-dimensional dataset like network traffic data. In reality, however, it is not an easy task to effectively perform the feature selection despite the availability of the existing selection techniques. From our initial experiments, we observed that the existing selection techniques produce different sets of features even under the same condition (e.g., a static size for the resulted set). In addition, individual selection techniques perform inconsistently, sometimes showing better performance but sometimes worse than others, thereby simply relying on one of them would be risky for building models using the selected features. More critically, it is demanding to automate the selection process, since it requires laborious efforts with intensive analysis by a group of experts otherwise. In this article, we explore challenges in the automated feature selection with the application of network anomaly detection. We first present our ensemble approach that benefits from the existing feature selection techniques by incorporating them, and one of the proposed ensemble techniques based on greedy search works highly consistently showing comparable results to the existing techniques. We also address the problem of when to stop to finalize the feature elimination process and present a set of methods designed to determine the number of features for the reduced feature set. Our experimental results conducted with two recent network datasets show that the identified feature sets by the presented ensemble and stopping methods consistently yield comparable performance with a smaller number of features to conventional selection techniques.

Automatic Model Selection For Anomaly Detection

A many-objective integrated evolutionary algorithm for feature selection in anomaly detection

A New Anomaly Detection Method Based on IGTE and IGFE.

Unsupervised Model Selection for Time-series Anomaly Detection

Accurate Anomaly Detection Leveraging Knowledge-enhanced GAT

Automated Feature Selection for Anomaly Detection in Network Traffic Data

An Empirical Analysis of Anomaly Detection Methods for Multivariate Time Series

An Anomaly Detection Algorithm Selection Service for IoT Stream Data Based on Tsfresh Tool and Genetic Algorithm

Robust Regression for Anomaly Detection.

Rapid Deployment of Anomaly Detection Models for Large Number of Emerging KPI Streams.

Borderline SMOTE Algorithm and Feature Selection-Based Network Anomalies Detection Strategy

An Ensemble Approach toward Automated Variable Selection for Network Anomaly Detection

Automated financial time series anomaly detection via curiosity-guided exploration and self-imitation learning

Anomaly Detection Model for Large-scale Industrial Systems Using Transfer Entropy and Graph Attention Network

An Effective Method for Computer Network Anomaly Detection

Enhancing Unsupervised Anomaly Detection with Score-Guided Network

A Graph Embedded In Graph Framework With Dual-sequence Input For Efficient Anomaly Detection of Complex Equipment Under Insufficient Samples

Refining the Optimization Target for Automatic Univariate Time Series Anomaly Detection in Monitoring Services

Metaheuristic and Data Mining Algorithms-based Feature Selection Approach for Anomaly Detection

Develop End-to-End Anomaly Detection System

An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series