SBE '—' A Precise Shellcode Detection Engine Based on Emulation and Support Vector Machine

Yonggan Hou,J. W. Zhuge,Dan Xin,Wenya Feng

DOI: https://doi.org/10.1007/978-3-319-06320-1_13

2014-01-01

Abstract:An important method of detecting zero-day attacks is to identify the shellcode which is usually taken as part of the attacks. However, the detection range is always restricted, for existent emulation based detection techniques only take several features that are observed when shellcode is emulated. In this paper, we propose a new shellcode detection algorithm based on emulation and Support Vector Machine(SVM). One of the most prominent advantages is that by means of emulating, we can get the real executed path which includes key features to identify shellcode e.g. loop, xor, GetPC etc. Moreover, by recording aforementioned features and training them with SVM, we can rely on general features to detect shellcode rather than on specific features. In addition, we build a complete shellcode data set so that other researchers can focus on detection algorithms. We have implemented a prototype system named SBE on Ubuntu/Amd-64 and tested our algorithm with various kinds of shellcode. Experiment shows that the proposed algorithm has a better detection rate than Libemu and could effectively detect all x86 shellcode with very few false positives.