Abstract:Large Language Models for Code (LLMs4Code) have been found to exhibit outstanding performance in the software engineering domain, especially the remarkable performance in coding tasks. However, even the most advanced LLMs4Code can inevitably contain incorrect or outdated code knowledge. Due to the high cost of training LLMs4Code, it is impractical to re-train the models for fixing these problematic code knowledge. Model editing is a new technical field for effectively and efficiently correcting erroneous knowledge in LLMs, where various model editing techniques and benchmarks have been proposed recently. Despite that, a comprehensive study that thoroughly compares and analyzes the performance of the state-of-the-art model editing techniques for adapting the knowledge within LLMs4Code across various code-related tasks is notably absent. To bridge this gap, we perform the first systematic study on applying state-of-the-art model editing approaches to repair the inaccuracy of LLMs4Code. To that end, we introduce a benchmark named CLMEEval, which consists of two datasets, i.e., CoNaLa-Edit (CNLE) with 21K+ code generation samples and CodeSearchNet-Edit (CSNE) with 16K+ code summarization samples. With the help of CLMEEval, we evaluate six advanced model editing techniques on three LLMs4Code: CodeLlama (7B), CodeQwen1.5 (7B), and Stable-Code (3B). Our findings include that the external memorization-based GRACE approach achieves the best knowledge editing effectiveness and specificity (the editing does not influence untargeted knowledge), while generalization (whether the editing can generalize to other semantically-identical inputs) is a universal challenge for existing techniques. Furthermore, building on in-depth case analysis, we introduce an enhanced version of GRACE called A-GRACE, which incorporates contrastive learning to better capture the semantics of the inputs.

Learning Domain-Specific Edit Operations from Model Repositories with Frequent Subgraph Mining

Mining Frequent Structures in Conceptual Models

Process collaboration based on automatic extraction of local ontology

A Maximal Common Subgraph Based Method for Process Retrieval

Towards Synthetic Trace Generation of Modeling Operations using In-Context Learning Approach

Analysing Time-Stamped Co-Editing Networks in Software Development Teams using git2net

Change Your History: Learning from Event Logs to Improve Processes

Exploring Model-Based Repositories for A Broad Range of Industrial Applications and Challenges

Model Editing for LLMs4Code: How Far are We?

git2net - Mining Time-Stamped Co-Editing Networks from Large git Repositories

Mining Frequent Patterns in Process Models

O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing

Collaboration Miner: Discovering Collaboration Petri Nets (Extended Version)

Generative retrieval-augmented ontologic graph and multi-agent strategies for interpretive large language model-based materials design

Adaptive Interaction Modeling via Graph Operations Search

Exploring In-Context Learning Capabilities of Foundation Models for Generating Knowledge Graphs from Text

Learning to Represent Edits

Automated Enterprise Architecture Model Mining

Automatic Discovery of Multi-perspective Process Model using Reinforcement Learning

Learning Structural Edits via Incremental Tree Transformations

Efficient Mining of Frequent Subgraphs in the Presence of Isomorphism