A Novel Hybrid Method for Polymorphic Worm Detection

Pan Xiaohui,Zhang Xiaosong,Chen Ting
DOI: https://doi.org/10.1109/ebiss.2009.5137885
2009-01-01
Abstract:Since worms have become a major threat of cybersecurity, several detection approaches have been proposed to detect them. However, attackers have exploited state-of-the-art techniques to evade these detection systems, such as polymorphism and metamorphism, making existing systems ineffective. In this paper, we propose a hybrid method for the detection of polymorphic worms. It uses improved Reverse Sequential Hypothesis Testing (RSHT) to detect portscans which are routinely used to find vulnerable hosts to compromise. Then a CPU emulator is used to execute every possible instruction sequence in suspicious traffic and determine whether it is an exploit code. We implemented a prototype and tested it using real polymorphic worms. Initial experimental results show that our approach is effective with high accuracy.
What problem does this paper attempt to address?