Abstract:To evade security detection, hackers always add a deceptive packer outside of the original malicious codes. The coexistence of original unpacked samples and packed samples of same family needs special attention in malware detection. The features of packed malware are changed by the packer, which would disturb the prediction results of malware classifier. The state-of-the-art studies of malware detection mainly focus on whether the malware is packed, or which type of packer is used. However, the ability of detecting the family of packed malware is still insufficient. Motivated by the above challenges, a novel packed malware family classification framework called MaliCage is proposed. The goal of the framework is to classify packed malware accurately. MaliCage consists of three core modules: packer detector, malware classifier, and a packer generative adversarial network (GAN). The packer detector is used as the pre-step of the framework to identify whether malware is packed. After dis-tinguishing the packed samples, the dynamic features extracted from the sandbox are fitted to the malware classifier based on deep neural networks (DNN). The malware classifier can classify unpacked and packed malware simultaneously. Furthermore, the packer GAN generates fake packed samples to alleviate the under -fitting of the malware classifiers. We built a single-packer dataset and a multi-packer dataset to evaluate the framework. In the single-packer experiment, 10 classes of malware samples packed by UPX were examined objectively. The accuracy of the malware classifier when using only real packed samples was 91.66%. After introducing fake packed samples generated by packer GAN, the accuracy of the packed malware classifier could reach 97.8%. In the multi-packer scenario, our method can also accurately classify benign programs, unpacked malware and malware packed by several common packers. The validation results show that MaliCage can not only effectively solve the impacts of packed malware on machine learning model, but also improve the detection accuracy.

MLxPack: Investigating the Effects of Packers on ML-based Malware Detection Systems Using Static and Dynamic Traits

Assessing the Impact of Packing on Machine Learning-Based Malware Detection and Classification Systems

Analysis of machine learning approaches to packing detection

An Efficient Multi-Step Framework for Malware Packing Identification

Efficient and Robust Malware Detection Based on Control Flow Traces Using Deep Neural Networks

Sensitive System Calls Based Packed Malware Variants Detection Using Principal Component Initialized MultiLayers Neural Networks

Malware Analysis Using Machine Learning and Deep Learning Techniques

Application of String Kernel Based Support Vector Machine for Malware Packer Identification

Packer Identification Using Hidden Markov Model

Research of dynamic general unpacking methods for Windows malware

Efficient Malware Packer Identification Using Support Vector Machines with Spectrum Kernel

Detection of Malicious Software by Analyzing Distinct Artifacts Using Machine Learning and Deep Learning Algorithms

Evaluation of Machine Learning Algorithms for Malware Detection

MaliCage: A packed malware family classification framework based on DNN and GAN

Unpacking Techniques and Tools in Malware Analysis

A fine-grained classification approach for the packed malicious code

Decoding the Secrets of Machine Learning in Malware Classification: A Deep Dive into Datasets, Feature Extraction, and Model Performance

Malware Detection Approach Based on Artifacts in Memory Image and Dynamic Analysis

Investigating Feature and Model Importance in Android Malware Detection: An Implemented Survey and Experimental Comparison of ML-Based Methods

Image Recognition-based Deep Neural Network for Packed Malware Detection