Abstract:Machine Learning models have many potentially beneficial applications in education settings, but a key barrier to their development is securing enough data to train these models. Labelling educational data has traditionally relied on highly skilled raters using complex, multi-class rubrics, making the process expensive and difficult to scale. An alternative, more scalable approach could be to use non-expert crowdworkers to evaluate student work, however, maintaining sufficiently high levels of accuracy and inter-rater reliability when using non-expert workers is challenging. This paper reports on two experiments investigating using non-expert crowdworkers and comparative judgement to evaluate complex student data. Crowdworkers were hired to evaluate student responses to open-ended reading comprehension questions. Crowdworkers were randomly assigned to one of two conditions: the control, where they were asked to decide whether answers were correct or incorrect (i.e., a categorical judgement), or the treatment, where they were shown the same question and answers, but were instead asked to decide which of two candidate answers was more correct (i.e., a comparative/preference-based judgement). We found that using comparative judgement substantially improved inter-rater reliability on both tasks. These results are in-line with well-established literature on the benefits of comparative judgement in the field of educational assessment, as well as with recent trends in artificial intelligence research, where comparative judgement is becoming the preferred method for providing human feedback on model outputs when working with non-expert crowdworkers. However, to our knowledge, these results are novel and important in demonstrating the beneficial effects of using the combination of comparative judgement and crowdworkers to evaluate educational data.

Adaptive Comparative Judgment as a Tool for Assessing Open-ended Design Problems and Model Eliciting Activities

Designing a video-stimulated questionnaire about teachers’ adaptive expertise in interdisciplinary mathematics and science teaching

Examining the Effects of a Real-Time, Knowledge-Aware Tool for Academic Writing Assessment

Assessing Students' Computational Thinking In A Learning By Modeling Environment

Adaptivity or agency? Educational technology design for conceptual learning of materials science

Preliminary Study on Students’ Experiences in Design-Based Interdisciplinary Learning

Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement

Modification and validation of the mixed‐format Engineering Concept Assessment for middle school students using many‐facet Rasch measurement

A systematic literature review of student evaluation of peer exemplars and implications for design, Technology, and Engineering Learning

Opportunities for Adaptive Experiments to Enable Continuous Improvement in Computer Science Education

Tools for Project-Based Active Learning of Amorphous Systems Design: Scenario Prototyping and Cross Team Peer Evaluation

An Investigation of Engineering Design Cognition and Achievement in Primary School

Assessment and Evaluation in Active Learning Implementations: Introducing the Engineering Education Active Learning Maturity Model

A Comparative Analysis of Student Perceptions of Recommendations for Engagement in Design Processes

Comparative Study of Engineering Design Project Assessment Rubrics to Address the Washington Accord’s Complexity Attributes

Comparative Design-Based Research: How Afterschool Programs Impact Learners' Engagement with a Video Game Codesign

Supporting Engineering Design Ideas with Science and Mathematics: A Case Study of Middle School Life Science Students

Utilizing Evidence-Centered Design to Develop Assessments: A High School Introductory Computer Science Course

Using Continuous Peer Evaluation in Team-Based Engineering Capstone Projects: A Case Study

Investigating the impacts of differentiated stimulus materials in a learning by evaluating activity

Assessing Learning of Computer Programming Skills in the Age of Generative Artificial Intelligence.