Abstract:CodeLLMs have demonstrated remarkable advancements in software engineering tasks. However, while these models can generate functionally correct code, they often produce code that is inefficient in terms of runtime. This inefficiency is particularly problematic in resource-constrained environments, impacting software performance and sustainability. Existing approaches for optimizing code efficiency for CodeLLMs like SOAP and PIE exhibit certain limitations. SOAP requires a compatible execution environment and predefined test cases for iterative code modification, while PIE focuses on instruction tuning, improving efficiency but compromising correctness. These shortcomings highlight the need for a fine-tuning framework that optimizes both efficiency and correctness without relying on predefined test cases or specific execution environments. To bridge this gap, we introduce ACECode, a reinforcement learning-based fine-tuning framework that aligns CodeLLMs with dual objectives of efficiency and correctness. ACECode combines three key steps: (1) generating code with an actor CodeLLM, (2) calculating a training-free reward signal derived from code execution feedback for each generated code, and (3) optimizing the CodeLLM via Proximal Policy Optimization (PPO) algorithm. This reward signal enables joint assessment of efficiency and correctness without manual labeling. We evaluate ACECode by fine-tuning four SOTA (state-of-the-art) CodeLLMs and comparing their code with three baselines: original, instruction-tuned, and PIE-tuned CodeLLMs. Extensive experiment results suggest that \tool{} significantly improves the efficiency and correctness of generated code against all baselines for all CodeLLMs. Specifically, CodeLLMs fine-tuned with ACECode improve pass@1 by 1.84% to 14.51% and reduce runtime in 65% to 72% of cases compared to original CodeLLMs.

Co-Learning: Code Learning for Multi-Agent Reinforcement Collaborative Framework with Conversational Natural Language Interfaces

Experiential Co-Learning of Software-Developing Agents

Code-mixed LLM: Improve Large Language Models' Capability to Handle Code-Mixing through Reinforcement Learning from AI Feedback

LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvement

Improving Natural Language Capability of Code Large Language Model

Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents

CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models

Code-Style In-Context Learning for Knowledge-Based Question Answering

Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback

WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning

Unity is Strength: Collaborative LLM-Based Agents for Code Reviewer Recommendation

RLCoder: Reinforcement Learning for Repository-Level Code Completion

Verco: Learning Coordinated Verbal Communication for Multi-agent Reinforcement Learning

Evaluating Performance of LLaMA2 Large Language Model Enhanced by QLoRA Fine-Tuning for English Grammatical Error Correction.

Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach

ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Models

StepCoder: Improving Code Generation with Reinforcement Learning from Compiler Feedback

Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint

CoLadder: Supporting Programmers with Hierarchical Code Generation in Multi-Level Abstraction

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback

Course-Correction: Safety Alignment Using Synthetic Preferences