Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications

Stephen Burabari Tete

2024-06-17

Abstract:The advent of Large Language Models (LLMs) has revolutionized various applications by providing advanced natural language processing capabilities. However, this innovation introduces new cybersecurity challenges. This paper explores the threat modeling and risk analysis specifically tailored for LLM-powered applications. Focusing on potential attacks like data poisoning, prompt injection, SQL injection, jailbreaking, and compositional injection, we assess their impact on security and propose mitigation strategies. We introduce a framework combining STRIDE and DREAD methodologies for proactive threat identification and risk assessment. Furthermore, we examine the feasibility of an end-to-end threat model through a case study of a custom-built LLM-powered application. This model follows Shostack's Four Question Framework, adjusted for the unique threats LLMs present. Our goal is to propose measures that enhance the security of these powerful AI tools, thwarting attacks, and ensuring the reliability and integrity of LLM-integrated systems.

Cryptography and Security,Software Engineering

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the new cybersecurity challenges introduced by large - language models (LLMs) in various applications. Specifically, the paper focuses on how to perform threat modeling and risk analysis on LLM - driven applications to identify and evaluate potential security threats and propose corresponding mitigation strategies. The paper mainly explores the following types of attacks and their impacts on application security: 1. **Data Poisoning Attack**: Manipulating training data to influence the model's behavior, which may lead to misclassification or biased output by the model. 2. **Prompt Injection Attacks**: Using malicious input to deceive the model into generating unexpected results. 3. **SQL Injection Attacks**: Through carefully designed prompts or inputs, forcing the LLM to generate or execute harmful SQL code, endangering data integrity and confidentiality. 4. **Jailbreaking Attacks**: Breaking through the LLM's restriction measures, making it deviate from the predetermined operating parameters. 5. **Compositional Injection Attacks**: Embedding multi - layer complex instructions in seemingly harmless prompts to achieve the purpose of unauthorized influence on the model's behavior. 6. **Insecure Output Handling**: Directly handling the output of the LLM without verification, which may lead to security problems such as cross - site request forgery (CSRF), server - side request forgery (SSRF), and cross - site scripting (XSS). To address these threats, the paper proposes a framework that combines the STRIDE and DREAD methodologies for proactive identification and evaluation of threats. In addition, through a case study, the paper also shows how to design an end - to - end threat model for a custom - built LLM - Doctor application. Ultimately, the paper aims to provide a set of measures to enhance the security of LLM - integrated systems, prevent attacks, and ensure the reliability and integrity of the systems.

Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications

Exploring Vulnerabilities and Threats in Large Language Models: Safeguarding Against Exploitation and Misuse

ThreatModeling-LLM: Automating Threat Modeling using Large Language Models for Banking System

Exploring Vulnerabilities and Protections in Large Language Models: A Survey

Large Language Models and Security

A Comprehensive Survey of Attack Techniques, Implementation, and Mitigation Strategies in Large Language Models

Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices

Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities

Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)

Recent Advances in Attack and Defense Approaches of Large Language Models

The Ethics of Interaction: Mitigating Security Threats in LLMs

Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposal

Attacks on Third-Party APIs of Large Language Models

Large Language Model Supply Chain: Open Problems From the Security Perspective

Large Language Models in Cybersecurity: State-of-the-Art

Adversarial attacks and defenses for large language models (LLMs): methods, frameworks & challenges

Jailbreaking and Mitigation of Vulnerabilities in Large Language Models

Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks

Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerations

Security and Privacy Challenges of Large Language Models: A Survey