Abstract:Since the beginning of the 21st century, the use of cloud computing has increased rapidly, and it currently plays a significant role among most organizations' information technology (IT) infrastructure. Virtualization technologies, particularly virtual machines (VMs), are widely used and lie at the core of cloud computing. While different operating systems can run on top of VM instances, in public cloud environments the Linux operating system is used 90% of the time. Because of their prevalence, organizational Linux-based virtual servers have become an attractive target for cyber-attacks, mainly launched by sophisticated malware designed at causing harm, sabotaging operations, obtaining data, or gaining financial profit. This has resulted in the need for an advanced and reliable unknown malware detection mechanism for Linux cloud-based environments. Antivirus software and today's even more advanced malware detection solutions have limitations in detecting new, unseen, and evasive malware. Moreover, many existing solutions are considered untrusted, as they operate on the inspected machine and can be interfered with, and can even be detected by the malware itself, allowing malware to evade detection and cause damage. In this paper, we propose Deep-Hook, a trusted framework for unknown malware detection in Linux-based cloud environments. Deep-Hook hooks the VM's volatile memory in a trusted manner and acquires the memory dump to discover malware footprints while the VM operates. The memory dumps are transformed into visual images which are analyzed using a convolutional neural network (CNN) based classifier. The proposed framework has some key advantages, such as its agility, its ability to eliminate the need for features defined by a cyber domain expert, and most importantly, its ability to analyze the entire memory dump and thus to better utilize the existing indication it conceals, thus allowing the induction of a more accurate detection model. Deep-Hook was evaluated on widely used Linux virtual servers; four state-of-the-art CNN architectures; eight image resolutions; and a total of 22,400 volatile memory dumps representing the execution of a broad set of benign and malicious Linux applications. Our experimental evaluation results demonstrate Deep-Hook's ability to effectively, efficiently, and accurately detect and classify unknown malware (even evasive malware like rootkits), with an AUC and accuracy of up to 99.9%.

HODetector: the Hidden Objects Detection Based on Static Semantic Information Library Outside Virtual Machine

Lightweight Intrusion Detection of Rootkit with VMI-Based Driver Separation Mechanism.

Virtual Mac Spoofing Detection Through Deep Learning

KVM-based Detection of Rootkit Attacks

Vmpmonitor: an Efficient Modularity Approach for Hidden Process Detection

A malware detection framework based on kolmogorov complexity

Kernel Intrusion Detection Method Based on Xen Hypervisor

Transparency and Semantics Coexist: When Malware Analysis Meets the Hardware Assisted Virtualization

Virtual-Machine-based Intrusion Detection on File-aware Block Level Storage.

HGDetector: A Hybrid Android Malware Detection Method Using Network Traffic and Function Call Graph

A Hybrid Detection Approach For Zero-Day Polymorphic Shellcodes

A Static Method for Detection of Information Theft Malware

Survey on Virtual Machine Environment Detection Methods

Inter-domain Communication Protocol for Real-time File Access Monitor of Virtual Machine

Virtual Wall: Filtering Rootkit Attacks to Protect Linux Kernel Functions.

Early Detection of In-Memory Malicious Activity based on Run-time Environmental Features

Deep-Hook: A trusted deep learning-based framework for unknown malware detection and classification in Linux cloud environments

Deep Learning-Based Vulnerability Detection and Mitigation in Virtualization Data Center

Enhancing Linux System Security: A Kernel-Based Approach to Fileless Malware Detection and Mitigation

Discovering Malicious Signatures in Software from Structural Interactions

A Novel Hybrid Method for Polymorphic Worm Detection