Abstract:The obfuscation detection technology is an important auxiliary means of malware detection. Also, for security practitioners, it can carry out automatic obfuscation detection before manual reverse analysis, which helps reverse engineers to perform reverse analysis more specifically. Existing obfuscation detection methods are mainly for Android applications and based on traditional machine learning, whose detection granularity is coarse, generality is poor, and the performance is not good enough. To address these issues, in this paper, we propose a function-level obfuscation detection method based on Graph Convolutional Networks for X86 assembly code and Android applications. Firstly, our method is function-level obfuscation detection, and we extract the Control Flow Graph (CFG) of each function as its feature, including the adjacency matrix and the basic block feature matrix. Secondly, we build a hybrid neural network model GCN-LSTM as our obfuscation detection model, which combines the Graph Convolutional Network (GCN) and the Long Short-Term Memory (LSTM). Finally, we conduct experiments using real-world open-source programs and compare results with baseline methods. For function-level detection, the accuracy of our method is 94.7575% for X86 assembly code and 98.9457% for Android applications, both of which are better than baseline methods. For APK-level detection, our method can almost completely detect the obfuscated APKs. Experimental results show that our method performs well for both X86 assembly code and Android applications and is superior to the baseline methods in both function-level detection and APK-level detection. Our research showcases a successful application of the Graph Convolutional Network and the Control Flow Graph on code obfuscation detection problems.

Deobfuscate Non-Returning Calls and Call-Stack Tampering in Instruction Traces

Parema: an Unpacking Framework for Demystifying VM-based Android Packers.

Targeting Infeasibility Questions on Obfuscated Codes

OCFI: Make Function Entry Identification Hard Again.

Khaos: The Impact of Inter-procedural Code Obfuscation on Binary Diffing Techniques

Deobfuscating Android Native Binary Code

Camouflage: Utility-Aware Obfuscation for Accurate Simulation of Sensitive Program Traces

A Method and Implementation of Control Flow Obfuscation Using SEH

Compiling Method for Obfuscation Technology Based on Crossing Control-Flow

Misleading Malware Similarities Analysis By Automatic Data Structure Obfuscation

Control Flow Obfuscation Scheme for LLVM Intermediate Languages

Pinpointing and Hiding Surprising Fragments in an Obfuscated Program.

Manufacturing Cheap, Resilient, and Stealthy Opaque Constructs.

Towards Generic Deobfuscation of Windows API Calls

Obfuscation as Instruction Decorrelation

How to Kill Symbolic Deobfuscation for Free; or Unleashing the Potential of Path-Oriented Protections

Function-level Obfuscation Detection Method Based on Graph Convolutional Networks

A Taxonomy of Obfuscating Transformations

Identification of Exception Return in Subroutine of Executable File

Extended Report on the Obfuscated Integration of Software Protections

Exploring Compiler Optimization Space for Control Flow Obfuscation