Abstract:Large Language Model (LLM) systems are inherently compositional, with individual LLM serving as the core foundation with additional layers of objects such as plugins, sandbox, and so on. Along with the great potential, there are also increasing concerns over the security of such probabilistic intelligent systems. However, existing studies on LLM security often focus on individual LLM, but without examining the ecosystem through the lens of LLM systems with other objects (e.g., Frontend, Webtool, Sandbox, and so on). In this paper, we systematically analyze the security of LLM systems, instead of focusing on the individual LLMs. To do so, we build on top of the information flow and formulate the security of LLM systems as constraints on the alignment of the information flow within LLM and between LLM and other objects. Based on this construction and the unique probabilistic nature of LLM, the attack surface of the LLM system can be decomposed into three key components: (1) multi-layer security analysis, (2) analysis of the existence of constraints, and (3) analysis of the robustness of these constraints. To ground this new attack surface, we propose a multi-layer and multi-step approach and apply it to the state-of-art LLM system, OpenAI GPT4. Our investigation exposes several security issues, not just within the LLM model itself but also in its integration with other components. We found that although the OpenAI GPT4 has designed numerous safety constraints to improve its safety features, these safety constraints are still vulnerable to attackers. To further demonstrate the real-world threats of our discovered vulnerabilities, we construct an end-to-end attack where an adversary can illicitly acquire the user's chat history, all without the need to manipulate the user's input or gain direct access to OpenAI GPT4. Our demo is in the link: https://fzwark.github.io/LLM-System-Attack-Demo/

LLM Security Guard for Code

Can LLMs Patch Security Issues?

LLMSecCode: Evaluating Large Language Models for Secure Coding

Can We Trust Large Language Models Generated Code? A Framework for In-Context Learning, Security Patterns, and Code Evaluations Across Diverse LLMs

LLMGuard: Guarding Against Unsafe LLM Behavior

SALLM: Security Assessment of Generated Code

Ocassionally Secure: A Comparative Analysis of Code Generation Assistants

LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks

Large Language Models and Code Security: A Systematic Literature Review

Generate and Pray: Using SALLMS to Evaluate the Security of LLM Generated Code

Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis

An Insight into Security Code Review with LLMs: Capabilities, Obstacles and Influential Factors

Code Security Vulnerability Repair Using Reinforcement Learning with Large Language Models

A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems

LLMs for Cyber Security: New Opportunities

CoSec: On-the-Fly Security Hardening of Code LLMs Via Supervised Co-Decoding

Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairs

Large Language Models for Secure Code Assessment: A Multi-Language Empirical Study

Is Your AI-Generated Code Really Safe? Evaluating Large Language Models on Secure Code Generation with CodeSecEval

"You still have to study" -- On the Security of LLM generated code