Abstract:Programming education at scale increasingly relies on automated feedback to help students learn to program. An important form of feedback is to point out semantic errors in student programs and provide hints for program repair. Such automated feedback depends essentially on solving the tasks of classification, localization and repair of semantic errors. Although there are datasets for the tasks, we observe that they do not have the annotations supporting all three tasks. As such, existing approaches for semantic error feedback treat error classification, localization and repair as independent tasks, resulting in sub-optimal performance on each task. Moreover, existing datasets either contain few programming assignments or have few programs for each assignment. Therefore, existing approaches often leverage rule-based methods and evaluate them with a small number of programming assignments. To tackle the problems, we first describe the creation of a new dataset COJ2022 that contains 5,914 C programs with semantic errors submitted to 498 different assignments in an introductory programming course, where each program is annotated with the error types and locations and is coupled with the repaired program submitted by the same student. We show the advantages of COJ2022 over existing datasets on various aspects. Second, we treat semantic error classification, localization and repair as dependent tasks, and propose a novel two-stage method ErrorCLR to solve them. Specifically, in the first stage we train a model based on graph matching networks to jointly classify and localize potential semantic errors in student programs, and in the second stage we mask error spans in buggy programs using information of error types and locations and train a CodeT5 model to predict correct spans. The predicted spans replace the error spans to form repaired programs. Experimental results show that ErrorCLR remarkably outperforms the comparative methods for all three tasks on COJ2022 and other public datasets. We also conduct a case study to visualize and interpret what is learned by the graph matching network in ErrorCLR. We have released the source code and COJ2022 at https://github.com/DaSESmartEdu/ErrorCLR.

ACcoding: A graph-based dataset for online judge programming

An Intelligent Online Judge System for Programming Training

The framework of a new online judge system for programming education.

An Intelligent Scheduling System for Large-Scale Online Judging.

Dataset of Program Source Codes Solving Unique Programming Exercises Generated by Digital Teaching Assistant

Programming Online Judge System

VisOJ: real-time visual learning analytics dashboard for online programming judge

CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks

Data Analysis of Online Judge System-Based Teaching Model.

Programming Knowledge Tracing: A Comprehensive Dataset and A New Model

How problem difficulty and order influence programming education outcomes in online judge systems

Programming grid: a computer-aided education system for programming courses based on online judge

TACO: Topics in Algorithmic COde Generation Dataset

Peking University Oneline Judge and Its Applications

Application of the Online Judge Technology in Programming Experimental Teaching

Hybrid Estimation for Open-Ended Questions with Early-Age Students' Block-Based Programming Answers.

Estimating Difficulty Levels of Programming Problems with Pre-trained Model

Educational Programming Systems for Learning at Scale

AIGCodeSet: A New Annotated Dataset for AI Generated Code Detection

ErrorCLR: Semantic Error Classification, Localization and Repair for Introductory Programming Assignments

How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data