Abstract:Tactics, Techniques, and Procedures (TTPs) outline the methods attackers use to exploit vulnerabilities. The interpretation of TTPs in the MITRE ATT&CK framework can be challenging for cybersecurity practitioners due to presumed expertise and complex dependencies. Meanwhile, advancements with Large Language Models (LLMs) have led to recent surge in studies exploring its uses in cybersecurity operations. It is, however, unclear how LLMs can be used in an efficient and proper way to provide accurate responses for critical domains such as cybersecurity. This leads us to investigate how to better use two types of LLMs: small-scale encoder-only (e.g., RoBERTa) and larger decoder-only (e.g., GPT-3.5) LLMs to comprehend and summarize TTPs with the intended purposes (i.e., tactics) of a cyberattack procedure. This work studies and compares the uses of supervised fine-tuning (SFT) of encoder-only LLMs vs. Retrieval Augmented Generation (RAG) for decoder-only LLMs (without fine-tuning). Both SFT and RAG techniques presumably enhance the LLMs with relevant contexts for each cyberattack procedure. Our studies show decoder-only LLMs with RAG achieves better performance than encoder-only models with SFT, particularly when directly relevant context is extracted by RAG. The decoder-only results could suffer low `Precision' while achieving high `Recall'. Our findings further highlight a counter-intuitive observation that more generic prompts tend to yield better predictions of cyberattack tactics than those that are more specifically tailored.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the difficult problems of understanding and interpreting TTP (Tactics, Techniques, and Procedures) in the field of network security. Specifically, the authors explore how to use large - language models (LLMs) to more effectively interpret and summarize TTPs in the MITRE ATT&CK framework and map them to specific network attack tactics. #### Main problems include: 1. **Complex TTP interpretation**: - Descriptions of TTPs in the MITRE ATT&CK framework are often complex and potentially ambiguous, causing different analysts to potentially draw different conclusions from the same description. - For example, the DLL search order hijacking technique can be associated with multiple tactics (such as privilege escalation, defense evasion, and persistence), but these associations are not always obvious. 2. **Application of LLMs in network security**: - Although LLMs have made significant progress in natural - language processing tasks, there is still uncertainty in their application in network security operations, especially in providing accurate responses. - Researchers need to explore how to use LLMs efficiently to interpret and map TTPs, especially when dealing with complex attack procedure descriptions. 3. **Comparison of different types of LLMs**: - Researchers hope to compare two types of LLMs: small - scale encoder - only models (such as RoBERTa) and large - scale decoder - only models (such as GPT - 3.5). - Specifically, they study the effect of supervised fine - tuning (SFT) on encoder models and the effect of retrieval - augmented generation (RAG) on decoder models. 4. **Improving the precision of decoder models**: - Decoder models perform well in recall but lack in precision. Researchers hope to improve the precision of decoder models by introducing techniques such as RAG without sacrificing their recall. 5. **Generic prompts vs. specific prompts**: - Research has found that generic prompts seem to provide more accurate and informative responses than specific prompts, which provides new ideas for future prompt design. ### Summary The paper experimentally compares the performance of different types of LLMs in interpreting TTPs, especially the differences between encoder models and decoder models under supervised fine - tuning and retrieval - augmented generation techniques. The research results not only reveal the potential of LLMs in the field of network security but also point out the direction for future improvement, especially in improving the precision of decoder models and optimizing prompt design.

Advancing TTP Analysis: Harnessing the Power of Large Language Models with Retrieval Augmented Generation

On the Uses of Large Language Models to Interpret Ambiguous Cyberattack Descriptions

From Text to MITRE Techniques: Exploring the Malicious Use of Large Language Models for Generating Cyber Attack Payloads

The Use of Large Language Models (LLM) for Cyber Threat Intelligence (CTI) in Cybercrime Forums

VTT-LLM: Advancing Vulnerability-to-Tactic-and-Technique Mapping through Fine-Tuning of Large Language Model

Tactics, Techniques, and Procedures (TTPs) in Interpreted Malware: A Zero-Shot Generation with Large Language Models

Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions

Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis

BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

A Comprehensive Overview of Large Language Models (LLMs) for Cyber Defences: Opportunities and Directions

TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models

The Best Defense is a Good Offense: Countering LLM-Powered Cyberattacks

CVE-driven Attack Technique Prediction with Semantic Information Extraction and a Domain-specific Language Model

LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testing

Cyber Knowledge Completion Using Large Language Models

Assessing Large Language Model’s knowledge of threat behavior in MITRE ATT&CK

RatGPT: Turning online LLMs into Proxies for Malware Attacks

Using Retriever Augmented Large Language Models for Attack Graph Generation

What You See Is Not Always What You Get: An Empirical Study of Code Comprehension by Large Language Models

Defending Large Language Models Against Attacks With Residual Stream Activation Analysis

Automated Progressive Red Teaming