Abstract:Context: The rapid evolution of Large Language Models (LLMs) has sparked significant interest in leveraging their capabilities for automating code review processes. Prior studies often focus on developing LLMs for code review automation, yet require expensive resources, which is infeasible for organizations with limited budgets and resources. Thus, fine-tuning and prompt engineering are the two common approaches to leveraging LLMs for code review automation. Objective: We aim to investigate the performance of LLMs-based code review automation based on two contexts, i.e., when LLMs are leveraged by fine-tuning and prompting. Fine-tuning involves training the model on a specific code review dataset, while prompting involves providing explicit instructions to guide the model's generation process without requiring a specific code review dataset. Methods: We leverage model fine-tuning and inference techniques (i.e., zero-shot learning, few-shot learning and persona) on LLMs-based code review automation. In total, we investigate 12 variations of two LLMs-based code review automation (i.e., GPT-3.5 and Magicoder), and compare them with the Guo et al.'s approach and three existing code review automation approaches (i.e., CodeReviewer, TufanoT5 and D-ACT). Results: The fine-tuning of GPT 3.5 with zero-shot learning helps GPT-3.5 to achieve 73.17%–74.23% higher EM than the Guo et al.'s approach. In addition, when GPT-3.5 is not fine-tuned, GPT-3.5 with few-shot learning achieves 46.38%–659.09% higher EM than GPT-3.5 with zero-shot learning. Conclusions: Based on our results, we recommend that (1) LLMs for code review automation should be fine-tuned to achieve the highest performance.; and (2) when data is not sufficient for model fine-tuning (e.g., a cold-start problem), few-shot learning without a persona should be used for LLMs for code review automation. Our findings contribute valuable insights into the practical recommendations and trade-offs associated with deploying LLMs for code review automation.

Fine-tuning Large Language Models to Improve Accuracy and Comprehensibility of Automated Code Review

LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning

Large Language Models for Code Analysis: Do LLMs Really Do Their Job?

Automating Patch Set Generation from Code Review Comments Using Large Language Models

Exploring the Capabilities of LLMs for Code Change Related Tasks

AI-powered Code Review with LLMs: Early Results

Prompting and Fine-tuning Large Language Models for Automated Code Review Comment Generation

Fine-tuning and prompt engineering for large language models-based code review automation

CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing

Automatically Correcting Large Language Models: Surveying the Landscape of Diverse Automated Correction Strategies

Robustness, Security, Privacy, Explainability, Efficiency, and Usability of Large Language Models for Code

Automated Commit Message Generation with Large Language Models: An Empirical Study and Beyond

Towards Reliable and Fluent Large Language Models: Incorporating Feedback Learning Loops in QA Systems

An Insight into Security Code Review with LLMs: Capabilities, Obstacles and Influential Factors

A Survey on Evaluation of Large Language ModelsJust Accepted

A Survey on Evaluating Large Language Models in Code Generation Tasks

Exploring Accuracy-Fairness Trade-off in Large Language Models

A Survey on Large Language Models for Code Generation

A Survey on Evaluation of Large Language Models

On the (In)Effectiveness of Large Language Models for Chinese Text Correction