Formatted Stateful Greybox Fuzzing of TLS Server

Fan Hu,Jiangan Ji,Hui Shu,Zheming Li,Tieming Liu,Chao Zhang

DOI: https://doi.org/10.1109/icst60714.2024.00022

2024-01-01

Abstract:The TLS protocol is one of the most crucial foundations for ensuring internet security. Consequently, vulnerabilities within the TLS protocol have a significant impact on the Internet security. This paper aims to explore more efficient methods of discovering vulnerabilities in the TLS protocol. Fuzzing stands out as one of the most important techniques for vulnerability discovery in the TLS protocol. To tackle the high complexity of the TLS protocol, stateful greybox fuzzers such as AFLnet have been introduced to enable stateful fuzzing of TLS servers. However, these mutation-based fuzzers often encounter chal-lenges in preserving the message format information during the mutation process, which can undermine the testing results. As a result, this paper proposes a novel approach that incorporates a formatted mutation strategy into the stateful greybox fuzzing process, with the aim of achieving more efficient mutation results. The evaluation process involves four mainstream fuzzers, with OpenSSL's TLS server serving as the target. The results demonstrate that the proposed method significantly enhances the quality of generated seeds, code coverage, and state coverage across all four fuzzers.