Abstract:Modern code completion engines, powered by large language models, have demonstrated impressive capabilities to generate functionally correct code based on surrounding context. As these tools are extensively used by millions of developers, it is crucial to investigate their security implications. In this work, we present INSEC, a novel attack that directs code completion engines towards generating vulnerable code. In line with most commercial completion engines, such as GitHub Copilot, INSEC assumes only black-box query access to the targeted engine, without requiring any knowledge of the engine's internals. Our attack works by inserting a malicious attack string as a short comment in the completion input. To derive the attack string, we design a series of specialized initialization schemes and an optimization procedure for further refinement. We demonstrate the strength of INSEC not only on state-of-the-art open-source models but also on black-box commercial services such as the OpenAI API and GitHub Copilot. On a comprehensive set of security-critical test cases covering 16 CWEs across 5 programming languages, INSEC significantly increases the likelihood of the considered completion engines in generating unsafe code by >50% in absolute, while maintaining the ability in producing functionally correct code. At the same time, our attack has low resource requirements, and can be developed for a cost of well under ten USD on commodity hardware.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to explore and address the security issues of modern code completion engines. Specifically, the paper proposes a new attack method called INSEC, which can guide black-box code completion engines to generate vulnerable code. The main problems and background of the paper are as follows: 1. **Widespread Application of Modern Code Completion Engines**: - Modern code completion engines, such as GitHub Copilot, are widely used in the daily work of millions of developers through large language models (LLMs), significantly improving programming efficiency. 2. **Urgency of Security Issues**: - Despite the powerful functionality of these tools, the code they generate may contain serious security vulnerabilities. Existing research indicates that even under normal circumstances, code generated by LLMs may contain dangerous security flaws. More concerning is that the frequency of generating vulnerabilities significantly increases when these models are subjected to poisoning attacks. 3. **Limitations of Existing Attacks**: - Current attack methods usually require in-depth knowledge of the model's internal structure, training data, etc., which is difficult to achieve in practical applications, especially for already deployed commercial services like GitHub Copilot. 4. **Characteristics of INSEC Attack**: - The INSEC attack assumes that the attacker only has black-box access, meaning they can only query the completion engine and obtain outputs without understanding or controlling the internal details of the engine. - The attack is implemented by inserting malicious comment strings into the code completion input. These strings are carefully designed and optimized to guide the completion engine to generate insecure code. 5. **Evaluation and Impact**: - The paper validates the effectiveness of INSEC through a series of experiments, including attacks on open-source models and commercial services. The results show that INSEC can significantly increase the probability of generating insecure code while maintaining functional correctness, raising serious concerns about the security of modern code completion engines. In summary, this paper attempts to reveal the potential security risks of modern code completion engines by proposing the INSEC attack method and calls for further research and development of corresponding mitigation measures.

Practical Attacks against Black-box Code Completion Engines

An LLM-Assisted Easy-to-Trigger Backdoor Attack on Code Completion Models: Injecting Disguised Vulnerabilities against Strong Detection

Security Attacks on LLM-based Code Completion Tools

CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models

Assessing Cybersecurity Vulnerabilities in Code Large Language Models

Enhancing Security of AI-Based Code Synthesis with GitHub Copilot via Cheap and Efficient Prompt-Engineering

What You See Is Not Always What You Get: An Empirical Study of Code Comprehension by Large Language Models

Codexity: Secure AI-assisted Code Generation

Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

"You still have to study" -- On the Security of LLM generated code

CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion

DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructions

Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Models

On Extracting Specialized Code Abilities from Large Language Models: A Feasibility Study

A Disguised Wolf Is More Harmful Than a Toothless Tiger: Adaptive Malicious Code Injection Backdoor Attack Leveraging User Behavior as Triggers

SecCodePLT: A Unified Platform for Evaluating the Security of Code GenAI

Backdooring Neural Code Search

SABER: Model-agnostic Backdoor Attack on Chain-of-Thought in Neural Code Generation

Is Your AI-Generated Code Really Safe? Evaluating Large Language Models on Secure Code Generation with CodeSecEval

Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants