Abstract:Malware is one of the most significant threats in today’s computing world since the number of websites distributing malware is increasing at a rapid rate. Malware analysis and prevention methods are increasingly becoming necessary for computer systems connected to the Internet. This software exploits the system’s vulnerabilities to steal valuable information without the user’s knowledge, and stealthily send it to remote servers controlled by attackers. Traditionally, anti-malware products use signatures for detecting known malware. However, the signature-based method does not scale in detecting obfuscated and packed malware. Considering that the cause of a problem is often best understood by studying the structural aspects of a program like the mnemonics, instruction opcode, API Call, etc. In this paper, we investigate the relevance of the features of unpacked malicious and benign executables like mnemonics, instruction opcodes, and API to identify a feature that classifies the executable. Prominent features are extracted using Minimum Redundancy and Maximum Relevance (mRMR) and Analysis of Variance (ANOVA). Experiments were conducted on four datasets using machine learning and deep learning approaches such as Support Vector Machine (SVM), Naïve Bayes, J48, Random Forest (RF), and XGBoost. In addition, we also evaluate the performance of the collection of deep neural networks like Deep Dense network, One-Dimensional Convolutional Neural Network (1D-CNN), and CNN-LSTM in classifying unknown samples, and we observed promising results using APIs and system calls. On combining APIs/system calls with static features, a marginal performance improvement was attained comparing models trained only on dynamic features. Moreover, to improve accuracy, we implemented our solution using distinct deep learning methods and demonstrated a fine-tuned deep neural network that resulted in an F1-score of 99.1% and 98.48% on Dataset-2 and Dataset-3, respectively.

Structural Feature Based Anomaly Detection for Packed Executable Identification

Packet Payload Based Anomalous Network Intrusion Detection

A fine-grained classification approach for the packed malicious code

Happer - Unpacking Android Apps via a Hardware-Assisted Approach.

PackerGrind: An Adaptive Unpacking System for Android Apps.

Packer Identification Using Hidden Markov Model

Assessing the Impact of Packing on Machine Learning-Based Malware Detection and Classification Systems

MLxPack: Investigating the Effects of Packers on ML-based Malware Detection Systems Using Static and Dynamic Traits

Feature Engineering Using File Layout for Malware Detection

An Efficient Multi-Step Framework for Malware Packing Identification

Analysis of machine learning approaches to packing detection

Detection of Malicious Software by Analyzing Distinct Artifacts Using Machine Learning and Deep Learning Algorithms

A Novel Malware Detection Mechanism based on Features Extracted from Converted Malware Binary Images

2-SPIFF: a 2-stage packer identification method based on function call graph and file attributes

Anomaly Detection for malware identification using Hardware Performance Counters

Efficient and Robust Malware Detection Based on Control Flow Traces Using Deep Neural Networks

Malicious Threats Detection of Executable File

Precise system-wide concatic malware unpacking

Impact of PCA Feature Extraction Method used in Malware Detection for Security Enhancement

Study of a Hybrid Approach Towards Malware Detection in Executable Files