Abstract:To maintain communications confidentiality, security protocols are widely used in more and more network applications. Moreover, some malwares even leverage these kinds of protocols to evade inspection by IDS. Most security protocols are designed and verified by formalized methods; however, observation shows that protocol implementations commonly contain flaws or vulnerabilities. Therefore, research on reverse engineering of security protocols can play an important role in improving the security of network applications, especially by providing another way to fight against malwares. Nevertheless, previous protocol reverse engineering technologies, which are based on analysis of network traces, encounter great challenges when the network messages transmitted between different protocol principals are encrypted. This paper proposes a taint analysis based method, which aims to infer the message format from dynamic execution of security protocol applications. The proposed approach is based on the observation that the process of message parsing in cryptographic protocol applications reveals rich information about the hierarchical structures and semantics of their messages. Hence, by observing calls to library function and instruction execution in network programs, the proposed approach can reverse derive large amount of information about their protocol, such as message format and protocol model, even the communication is encrypted. Experiments show that the reverse analysis results not only accurately identify message fields, but also unveil the structure of the encrypted message fields.

Specifying and Checking Network Protocol Based on TLA

Protocol Security Testing with SPIN and TTCN-3

A new method for analyzing and improving security protocols

A Formal TLS Handshake Model in LNT

Formal Modeling and Analyzing Kerberos Protocol

Protocol vulnerability detection based on network traffic analysis and binary reverse engineering

Integrated TCP/IP protocol software testing for vulnerability detection

Network Protocol System Monitoring - A Formal Approach with Passive Testing

Select Accept Dynamic Logic

A model-guided symbolic execution approach for network protocol implementations and vulnerability detection

Network Security Testing Based on Protocol Implements

Finding Security Vulnerabilities in Network Protocol Implementations

Verification Tool for Security Protocol Based on GSPM

Automatic Vulnerability Checking of IEEE 802.16 WiMAX Protocols through TLA+

From Wireless Sensor Networks to Wireless Body Area Networks: Formal Modeling and Verification on Security Using PAT.

SAT-based Lazy Formal Analysis Method for Security Protocols

Reverse analysis of secure communication protocol based on taint analysis

An Enhanced Passive Testing Tool for Network Protocols

A Network Protocol Reverse Engineering Method Based On Dynamic Taint Propagation Similarity

Reverse Analysis and Vulnerability Detection for Network System Software

Modeling and Verifying Cryptographic Protocols Using SPIN