Abstract:The attack that occurred recently involved the utilization of malicious software, commonly referred to as malware, along with advanced techniques such as machine learning, specifically deep learning, code transformation, and polymorphism. This makes it harder for cyber experts to detect malware using traditional analysis methods. In view of the low accuracy and high false positive rate of traditional malware detection methods, this research proposes a fine-tuned deep learning model with a novel dataset that is compared with ANN, Support Vector Machine (SVM), random forest (RF), K-Nearest Neighbourhood (KNN) classifiers. Converting a malware code into an image could allow users to effectively identify the presence of malware, even if the original code is modified by the creator. This is due to the fact that the attributes of images remain unchanged, allowing for reliable identification. So, researchers used deep learning technology to detect malware, like detecting malaria from red blood cells. The deep learning model found that a more detailed analysis of malware data sets, focusing on RGB and greyscale images, is needed. These data sets currently rely on publicly available data, but the accuracy of the traditional model could be a lot higher. It also produces many false positive results. The main goal is to create a new data set and model using malware images to identify and categorize malware using deep learning without relying on existing image detection and transfer learning models. The researcher adjusted different hyper parameters, like the number of neurons, filters, stride, hidden units, layers, learning rate, batch size, activation function, optimizer, and epochs. Identifying and correcting issues in models, improving their clarity, stability, and fairness, as well as debugging and monitoring them, is a challenging task. To eliminate this obstacle, assess the model's behaviour and performance by employing various methods such as logging, profiling, testing, visualization, and model clarity. To overcome the challenges posed by the electricity shutdown, we utilized Google's cloud-based GPU and Python 3.7 language to conduct the experiment and train the model. Kali Linux is an operating system that can automatically encrypt file systems and has a lower chance of crashing the system due to malware in a virtual sandbox. The researcher used techniques like early stopping and cross-validation to prevent over fitting and assess generalization in addition to monitoring and evaluating the model's behaviour and performance. The researcher used different methods like L1 and L2 regularization, dropout, and batch normalization to improve the model's performance and avoid over fitting. The binary portable executable (PE) malware dataset is collected from Kaggle, Malimg, Virusshare, Malvis, MS Big2015, and VX-underground and finally converted to greyscale and RGB images to create a novel dataset to fill the lack of image dataset. The raw dataset was then rescaled to a 128x128 greyscale and RGB (red, green, blue) image and flattened to 1024-byte vector images input into convolution neural network (CNN) interpolation to extract features for malware detection. The newly acquired dataset is utilized as an input for the innovative DL algorithms in order to create a tailor-made model capable of accurately predicting malware. The findings in this customized CNN model by fine-tuning hyper-parameters with the three-channel RGB dataset outperformed a greyscale dataset with an accuracy of 98.7% and error rate of 1.3% in current malware compared with other artificial neural network (ANN), ML algorithms such as Support Vector Machine (SVM), K-Nearest Neighbour (KNN), and also Random Forest (RF) models. The proposed approach is compatible with all operating systems (Windows, Linux, and Mac) and can identify different types of malwares, such as packed, polymorphic, obfuscated, metamorphic, or variations of a malware family. There are some limitations to the shortage of malware image datasets and computational costs for fine-tuning hyper parameters in the whole model. Furthermore, the proposed approach detects malware and groups it into families without using common techniques like disassembly, decompilation, de-obfuscation, or running malicious code in a virtual environment.

MaliCage: A packed malware family classification framework based on DNN and GAN

Black-Box Adversarial Attacks Against Deep Learning Based Malware Binaries Detection with GAN

An Efficient Multi-Step Framework for Malware Packing Identification

MLxPack: Investigating the Effects of Packers on ML-based Malware Detection Systems Using Static and Dynamic Traits

Efficient and Robust Malware Detection Based on Control Flow Traces Using Deep Neural Networks

A Malware Family Classification Method Based on the Point Cloud Model DGCNN

Deep hybrid approach with sequential feature extraction and classification for robust malware detection

Assessing the Impact of Packing on Machine Learning-Based Malware Detection and Classification Systems

ApkClassiFy: Identification and Classification of Packed Android Malicious Applications.

A fine-grained classification approach for the packed malicious code

S-DCNN: stacked deep convolutional neural networks for malware classification

Task-Aware Meta Learning-based Siamese Neural Network for Classifying Obfuscated Malware

Image-Based Malware Classification Using Ensemble of CNN Architectures (IMCEC)

A Malware Classification Method Based on the Capsule Network

Development of a Deep Stacked Ensemble With Process Based Volatile Memory Forensics for Platform Independent Malware Detection and Classification

Packer Identification Using Hidden Markov Model

CNN‐ and GAN‐based classification of malicious code families: A code visualization approach

BenchMFC: A Benchmark Dataset for Trustworthy Malware Family Classification under Concept Drift

Analysis of machine learning approaches to packing detection

Malware Family Classification Using Lstm With Attention

Malware Detection using Deep Learning (DL)