AGAE: Unsupervised Anomaly Detection for Encrypted Malicious Traffic

Hao Wang,Ye Wang,Zhaoquan Gu,Yan Jia
DOI: https://doi.org/10.1007/978-981-97-7241-4_28
2024-01-01
Abstract:Nowadays, In order to protect the privacy and security of network users, network traffic is extensively encrypted. However, encrypted traffic can also be exploited by attackers to conceal their malicious activities. Moreover, existing approaches heavily rely on supervised learning and labeled datasets. Thus, effectively detecting malicious traffic with limited data remains an unresolved issue. In this paper, we propose AGAE, an unsupervised anomaly detection system for detecting malicious traffic, based on the Attribute Graph AutoEncoder we designed. We innovatively analyze the convergence and reusability of network attacks, and design AGAE by using these two characteristics. We conduct extensive experiments to evaluate the performance of AGAE. The experimental results illustrate that the AGAE achieves average AUC of 0.961. And the average F1 achieves 0.974, which outperform the state-of-the-art methods. In particular, AGAE has stronger detection capabilities against traditional brute force attacks and encrypted flooding traffic.
What problem does this paper attempt to address?