Abstract:With the growing use of information technology in all life domains, hacking has become more negatively effective than ever before. Also with developing technologies, attacks numbers are growing exponentially every few months and become more sophisticated so that traditional IDS becomes inefficient detecting them. This paper proposes a solution to detect not only new threats with higher detection rate and lower false positive than already used IDS, but also it could detect collective and contextual security attacks. We achieve those results by using Networking Chatbot, a deep recurrent neural network: Long Short Term Memory (LSTM) on top of Apache Spark Framework that has an input of flow traffic and traffic aggregation and the output is a language of two words, normal or abnormal. We propose merging the concepts of language processing, contextual analysis, distributed deep learning, big data, anomaly detection of flow analysis. We propose a model that describes the network abstract normal behavior from a sequence of millions of packets within their context and analyzes them in near real-time to detect point, collective and contextual anomalies. Experiments are done on MAWI dataset, and it shows better detection rate not only than signature IDS, but also better than traditional anomaly IDS. The experiment shows lower false positive, higher detection rate and better point anomalies detection. As for prove of contextual and collective anomalies detection, we discuss our claim and the reason behind our hypothesis. But the experiment is done on random small subsets of the dataset because of hardware limitations, so we share experiment and our future vision thoughts as we wish that full prove will be done in future by other interested researchers who have better hardware infrastructure than ours.

Advanced Network Security Analysis (ANSA) in Big Data Technology

Cyberattack Data Analysis in IoT Environments using Big Data

Big Data Architecture for Network Security

An Artificial Intelligence-based, Big Data-aware, Long-lasting Security Solution for the Internet of Things

Intelligent Threat Detection—AI-Driven Analysis of Honeypot Data to Counter Cyber Threats

Evaluation of Performance for Big Data Security Using Advanced Cryptography Policy

Attribute based honey encryption algorithm for securing big data: Hadoop distributed file system perspective

Exploring Wireshark For Network Traffic Analysis

Applying Hadoop for log analysis toward distributed IDS

An Exhaustive Research on the Application of Intrusion Detection Technology in Computer Network Security in Sensor Networks

Big data analysis and distributed deep learning for next-generation intrusion detection system optimization

Improvement of data analysis and protection using novel privacy-preserving methods for big data application

Research on Network Security Analysis Based on Big Data Technology Application

Security Analysis of Big Data on Internet of Things

Design of Computer Network Security Defense System Based on Big Data

Big Data Analysis Based Network Behavior Insight of Cellular Networks for Industry 4.0 Applications

Prevention of cyberattacks in WSN and packet drop by CI framework and information processing protocol using AI and Big Data

Auditing of hadoop log file for dynamic detection of threats using H-ISSM-MIM and convolutional neural network

Network Traffic Analysis:Hadoop Pig vs Typical MapReduce

An Analysis of the Domain Names Linked to the Important World Events using a Longitudinal Measurement Approach

Large-scale Data Modelling in Hive and Distributed Query Processing using MapReduce and Tez