Abstract:With the growing use of information technology in all life domains, hacking has become more negatively effective than ever before. Also with developing technologies, attacks numbers are growing exponentially every few months and become more sophisticated so that traditional IDS becomes inefficient detecting them. This paper proposes a solution to detect not only new threats with higher detection rate and lower false positive than already used IDS, but also it could detect collective and contextual security attacks. We achieve those results by using Networking Chatbot, a deep recurrent neural network: Long Short Term Memory (LSTM) on top of Apache Spark Framework that has an input of flow traffic and traffic aggregation and the output is a language of two words, normal or abnormal. We propose merging the concepts of language processing, contextual analysis, distributed deep learning, big data, anomaly detection of flow analysis. We propose a model that describes the network abstract normal behavior from a sequence of millions of packets within their context and analyzes them in near real-time to detect point, collective and contextual anomalies. Experiments are done on MAWI dataset, and it shows better detection rate not only than signature IDS, but also better than traditional anomaly IDS. The experiment shows lower false positive, higher detection rate and better point anomalies detection. As for prove of contextual and collective anomalies detection, we discuss our claim and the reason behind our hypothesis. But the experiment is done on random small subsets of the dataset because of hardware limitations, so we share experiment and our future vision thoughts as we wish that full prove will be done in future by other interested researchers who have better hardware infrastructure than ours.

Big Data Classification Using Belief Decision Trees: Application to Intrusion Detection

A Mixed Intrusion Detection System Utilizing K-means and Extreme Gradient Boosting

A Classification Tree Method Based on Belief Entropy for Evidential Data

An intrusion detection method based on decision tree

Multivariate Big Data Analysis for Intrusion Detection: 5 steps from the haystack to the needle

An Ensemble-Based Scalable Approach for Intrusion Detection Using Big Data Framework

An Ensemble Approach for Feature Selection and Classification in Intrusion Detection Using Extra-Tree Algorithm

Combining Naive Bayes and Decision Tree for Adaptive Intrusion Detection

Cyber Intrusion Detection Using Machine Learning Classification Techniques

Explainable Artificial Intelligence (XAI) to Enhance Trust Management in Intrusion Detection Systems Using Decision Tree Model

Study of Data Mining in Intrusion Detection

Big data analysis and distributed deep learning for next-generation intrusion detection system optimization

An improved anomaly detection model for IoT security using decision tree and gradient boosting

Decision Tree Applied in Web-Based Intrusion Detection System

Intrusion detection model using machine learning algorithm on Big Data environment

Comparative Analysis of Intrusion Detection Models using Big Data Analytics and Machine Learning Techniques

An empirical evaluation for the intrusion detection features based on machine learning and feature selection methods

A Hybrid Visualizing Approach Based on Adaptive Neuro-fuzzy Inference System, Decision Tree, and K-means for Intrusion Detection

A distributed platform for intrusion detection system using data stream mining in a big data environment

Towards Developing a Robust Intrusion Detection Model Using Hadoop–Spark and Data Augmentation for IoT Networks