Abstract:Intrusion detection system (IDS) provides an important basis for the network defense. Due to the development of the cloud computing and social network, massive amounts of data are generated, which inevitably brings much pressure to IDS. And therefore, it becomes crucial to efficiently divide the data into different classes over big data according to data features. Moreover, we can further determine whether one is normal behavior or not based on the classes information. Although the clustering approach based on K-means for IDS has been well studied, unfortunately directly using it in big data environment may suffer from inappropriateness. On the one hand, the efficiency of data clustering needs to be improved. On the other hand, differ from the classification, there is no unified evaluation indicator for clustering issue, and thus, it is necessary to study which indicator is more suitable for evaluating the clustering results of IDS. In this paper, we propose a clustering method for IDS based on Mini Batch K-means combined with principal component analysis. First, a preprocessing method is proposed to digitize the strings and then the data set is normalized so as to improve the clustering efficiency. Second, the principal component analysis method is used to reduce the dimension of the processed data set aiming to further improve the clustering efficiency, and then mini batch K-means method is used for data clustering. More specifically, we use K-means++ to initialize the centers of cluster in order to avoid the algorithm getting into the local optimum, in addition, we choose the Calsski Harabasz indicator so that the clustering result is more easily determined. Compared with the other methods, the experimental results and the time complexity analysis show that our proposed method is effective and efficient. Above all, our proposed clustering method can be used for IDS over big data environment.

Applying Hadoop for log analysis toward distributed IDS

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

A Distributedsystem of Log Analysis Based on Hadoop

Research on log data analysis technology based on improved Hadoop

Auditing of hadoop log file for dynamic detection of threats using H-ISSM-MIM and convolutional neural network

Big data analysis and distributed deep learning for next-generation intrusion detection system optimization

A Distributed Processing Method for Large Scale Logs

An Integrated Method for Anomaly Detection From Massive System Logs.

A parallel clustering algorithm for logs data based on Hadoop platform

Distributed Log Analysis on the Cloud Using MapReduce

Advanced Network Security Analysis (ANSA) in Big Data Technology

Clustering Approach Based On Mini Batch Kmeans For Intrusion Detection System Over Big Data

A Data Mining-based Approach for IDS Log Analysis

Big-data Analysis of Multi-Source Logs for Anomaly Detection on Network-Based System.

Network Log Analysis with SQL-on-Hadoop

Applying High-Performance Bioinformatics Tools for Outlier Detection in Log Data

Design and Implementation of Log Data Analysis Management System Based on Hadoop

Investigation of network traffic in geo-distributed data centers

Network Traffic Analysis:Hadoop Pig vs Typical MapReduce

Log Analysis For Network Attack Detection Using Deep Learning Models

Extending Isolation Forest for Anomaly Detection in Big Data via K-Means