Abstract:In an increasingly digitalized world, cybersecurity has emerged as a critical component of safeguarding sensitive information and infrastructure from malicious threats. The threat actors are often in line or even one step ahead of the defense, causing the increasing reliance of security teams on artificial intelligence while trying to detect zero-day attacks. However, most of the cybersecurity solutions based on artificial intelligence that can be found in the literature are trained and tested on reference datasets that are at least five or more years old, which gives a vague insight into their security performances. Moreover, they often tend to be designed as isolated, self-focused components. The aim of this paper is to design and implement a modular network intrusion detection architecture capable of simulating cyberattacks based on real-world scenarios while evaluating its defense capabilities. The architecture is designed as a full pipeline from real-time network data collection and transformation to threat-information presentation and visualization, with a pre-trained artificial intelligence module at its core. Well-known components like CICFlowMeter, Prometheus, and Grafana are used and modified to fit our data preparation and core modules to form the proposed architecture for real-world network traffic security monitoring. For the sake of cyberattack simulation, the proposed architecture is situated within a virtual environment, surrounded by the Kali Linux-based penetration simulation agent on one side and a vulnerable agent on the other. The intrusion detection artificial intelligence module is trained on the CICIDS-2017 dataset, and it is demonstrated using the proposed architecture that, despite being trained on an outdated dataset, the trained module is still effective in detecting sophisticated modern attacks. Two case studies are given to illustrate how modular architectures and virtual environments can be valuable tools to assess the security properties of artificial intelligence-based solutions through simulation in real-world scenarios.

Implementation and visualization of a netflow log data lake system for cyberattack detection using distributed deep learning

Cyberattack detection model using deep learning in a network log system with data visualization

Big data analysis and distributed deep learning for next-generation intrusion detection system optimization

Cyberattacks detection and analysis in a network log system using XGBoost with ELK stack

Real-time Distributed-Random-Forest-Based Network Intrusion Detection System Using Apache Spark

Log Analysis For Network Attack Detection Using Deep Learning Models

Fuzzy Local Information and Bhattacharya-Based C-Means Clustering and Optimized Deep Learning in Spark Framework for Intrusion Detection

Deep Learning for Malicious Flow Detection

Big-data Analysis of Multi-Source Logs for Anomaly Detection on Network-Based System.

ENCODE: Encoding NetFlows for Network Anomaly Detection

Machine-Learning-Based Online Distributed Denial-of-Service Attack Detection Using Spark Streaming

Distributed system anomaly detection using deep learning‐based log analysis

Smart city infrastructure protection: real-time threat detection employing online reservoir computing architecture

A NetFlow based flow analysis and monitoring system in enterprise networks

Network intrusion detection: An optimized deep learning approach using big data analytics

Learning to Detect: A Data-driven Approach for Network Intrusion Detection

Toward a monitoring and threat detection system based on stream processing as a virtual network function for big data

Modular deep learning-based network intrusion detection architecture for real-world cyber-attack simulation

A Real Time Deep Learning based Approach for Detecting Network Attacks

Towards Detecting Flooding DDOS Attacks Over Software Defined Networks Using Machine Learning Techniques

An Artificial Intelligence-based, Big Data-aware, Long-lasting Security Solution for the Internet of Things