Abstract:The capabilities of Large Language Models (LLMs) have significantly evolved, extending from natural language processing to complex tasks like code understanding and generation. We expand the scope of LLMs' capabilities to a broader context, using LLMs to execute code snippets to obtain the output. This paper pioneers the exploration of LLMs as code executors, where code snippets are directly fed to the models for execution, and outputs are returned. We are the first to comprehensively examine this feasibility across various LLMs, including OpenAI's o1, GPT-4o, GPT-3.5, DeepSeek, and Qwen-Coder. Notably, the o1 model achieved over 90% accuracy in code execution, while others demonstrated lower accuracy levels. Furthermore, we introduce an Iterative Instruction Prompting (IIP) technique that processes code snippets line by line, enhancing the accuracy of weaker models by an average of 7.22% (with the highest improvement of 18.96%) and an absolute average improvement of 3.86% against CoT prompting (with the highest improvement of 19.46%). Our study not only highlights the transformative potential of LLMs in coding but also lays the groundwork for future advancements in automated programming and the completion of complex tasks.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to explore the capability of large language models (LLMs) as code executors. Specifically, it systematically investigates the feasibility of LLMs directly executing code snippets for the first time and evaluates the performance of different models in this task. By introducing the Iterative Instruction Prompting (IIP) technique, the paper demonstrates how to improve the execution accuracy of weaker models. #### Main Objectives: 1. **Evaluate the performance of LLMs as code executors**: The paper evaluates the performance of various LLMs, including OpenAI's o1, GPT-4o, GPT-3.5, DeepSeek, and Qwen-Coder, in executing code. 2. **Propose the Iterative Instruction Prompting technique**: The paper introduces a new method—Iterative Instruction Prompting (IIP)—which improves the execution accuracy of models by processing code snippets line by line. 3. **Analyze influencing factors**: The paper analyzes the impact of code type, code snippet length, and computational complexity on model performance. 4. **Multi-language support**: The paper collects code snippets in both Chinese and English to evaluate the performance of LLMs in different language environments. #### Key Findings: - **Outstanding performance of OpenAI's o1 model**: The o1 model achieved over 90% accuracy in executing code. - **Effectiveness of the Iterative Instruction Prompting technique**: The IIP technique improved accuracy by an average of 7.22%, with a maximum improvement of 18.96%. - **Multi-language differences**: Most models performed better on English code snippets than on Chinese code snippets. - **Performance differences across problem types**: LLMs performed best on binary search problems and worst on bit manipulation and dynamic programming problems. Through these studies, the paper not only demonstrates the potential of LLMs in code execution but also lays the foundation for future automatic programming and complex task execution.

Large Language Models as Code Executors: An Exploratory Study

If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents

Large Language Models for Code Analysis: Do LLMs Really Do Their Job?

Steering Large Language Models between Code Execution and Textual Reasoning

Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study

Executing Natural Language-Described Algorithms with Large Language Models: An Investigation

On the Effectiveness of Large Language Models in Domain-Specific Code Generation

LLMs for Coding and Robotics Education

Planning-Driven Programming: A Large Language Model Programming Workflow

L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models

LLM4DS: Evaluating Large Language Models for Data Science Code Generation

CodeT5+: Open Code Large Language Models for Code Understanding and Generation

Enabling Programming Thinking in Large Language Models Toward Code Generation

The First Prompt Counts the Most! An Evaluation of Large Language Models on Iterative Example-based Code Generation

CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors

A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends

CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models

Should AI Optimize Your Code? A Comparative Study of Current Large Language Models Versus Classical Optimizing Compilers

CodeJudge: Evaluating Code Generation with Large Language Models