Abstract:The commonly used method for network intrusion is based on pattern matching systems, but these systems do not have high detection accuracy when facing large-scale high-speed network traffic environments. In addition, when facing a wide variety of network attacks, relying solely on a certain network security technology is difficult to ensure network security. Therefore, a distributed network intrusion prevention system based on the Spark framework and dynamic information security theory model was innovatively proposed to improve the detection efficiency of network intrusion and solve these issues. The architecture and functional structure of the network intrusion prevention system were constructed by improving the random forest algorithm and Spark. In addition, the dynamic network intrusion prevention system was designed on the basis of the Policy Protection Detection Response (P2DR) model and the Protection Detection Reaction Recovery (PDRR) model to cope with the diverse network attacks and make up for the lack of dynamic defense based on improved forest algorithm and Spark. The test results showed that the network intrusion prevention system based on the improved random forest algorithm and Spark had a maximum detection time of 13,593 ms, a minimum detection time of 13,318 ms, and an average detection time of 13,468 ms when the data were 6000 pieces. The average success rate of the dynamic network intrusion prevention system based on the dynamic information security theory model was 87.72%, the average detection rate was 71.68%, and the average false alarm rate was 17.23%. The F1 values corresponding to the improved random forest algorithm under 7 different attack types of data were 0.985, 0.983, 0.876, 0.843, 0.797, 0.983, and 0.890, respectively, which were significantly better than the comparison algorithm. It can be seen that the dynamic network intrusion prevention system based on the improved random forest algorithm, Spark, and the dynamic information security theoretical model, has good performance and can effectively detect network intrusions, providing technical support for solving network intrusion problems in reality. The contribution of the research is reflected in the improvement of the detection performance of network intrusion detection systems and the reduction of detection time and false alarm rates.

Design and application of real-time network abnormal traffic detection system based on Spark Streaming

Real-time Detection of Traffic Congestion Based on Trajectory Data

An Algorithm for Data Stream Speed Anomaly Detection Based on RCSW

Socialanalysis: A Real-Time Query And Mining System From Social Media Data Streams

DDoS Attack Detection Method Based on Network Abnormal Behavior in Big Data Environment

Anomaly Detection of Large Scale Network Based on Data Streams

Design and Implementation of Real Time Data Processing System Based on Spark Streaming

Big-data Analysis of Multi-Source Logs for Anomaly Detection on Network-Based System.

A Deep Random Forest Model on Spark for Network Intrusion Detection

A Distributed Network Intrusion Detection System for Distributed Denial of Service Attacks in Vehicular Ad Hoc Network

Design of distributed network intrusion prevention system based on Spark and P2DR models

Toward a monitoring and threat detection system based on stream processing as a virtual network function for big data

Research on the detection of abnormal traffic for multi-channel network

Internet Traffic Analysis Using Community Detection and Apache Spark

Anomaly Detection of Streaming Data Based on Deep Learning

Micro-cluster-based Online Network Abnormal Detection Method

Big data analysis and distributed deep learning for next-generation intrusion detection system optimization

Anomaly Intrusion Detection Based on Data Stream

Real-Time DDoS Attack Detection System Using Big Data Approach

Fuzzy Local Information and Bhattacharya-Based C-Means Clustering and Optimized Deep Learning in Spark Framework for Intrusion Detection

Distributed anomaly detection using concept drift detection based hybrid ensemble techniques in streamed network data