Abstract:Introduction: In the field of education, new technologies have enhanced the objectivity and scientificity of educational evaluation. However, concerns have been raised about the fairness of evaluators, such as artificial intelligence (AI) algorithms. This study aimed to assess college students' perceptions of fairness in educational evaluation scenarios through three studies using experimental vignettes. Methods: Three studies were conducted involving 172 participants in Study 1, 149 in Study 2, and 145 in Study 3. Different evaluation contexts were used in each study to assess the influence of evaluators on students' perception of fairness. Information transparency and explanations for evaluation outcomes were also examined as potential moderators. Results: Study 1 found that different evaluators could significantly influence the perception of fairness under three evaluation contexts. Students perceived AI algorithms as fairer evaluators than teachers. Study 2 revealed that information transparency was a mediator, indicating that students perceived higher fairness with AI algorithms due to increased transparency compared with teachers. Study 3 revealed that the explanation of evaluation outcomes moderated the effect of evaluator on students' perception of fairness. Specifically, when provided with explanations for evaluation results, the effect of evaluator on students' perception of fairness was lessened. Discussion: This study emphasizes the importance of information transparency and comprehensive explanations in the evaluation process, which is more crucial than solely focusing on the type of evaluators. It also draws attention to potential risks like algorithmic hegemony and advocates for ethical considerations, including privacy regulations, in integrating new technologies into educational evaluation systems. Overall, this study provides valuable theoretical insights and practical guidance for conducting fairer educational evaluations in the era of new technologies.

Teachers' Conceptions of Fairness in Classroom Assessment: An Empirical Study

Digital Module 33: Fairness in Classroom Assessment: Dimensions and Tensions

Classroom assessment fairness inventory: a new instrument to support perceived fairness in classroom assessment

Fairness in assessment practices in online education: Iranian University English teachers’ perceptions

Demographic biases and assessment fairness in classroom: insights from Iranian university teachers

Measuring Fairness and Justice in the Classroom: A Systematic Review of Instruments' Validity Evidence

Three Foci of Mathematic Teachers' Considerations in Evaluation of School Students' Achievements

Exploring student teachers’ perceptions of assessment ethics across university-based teacher education programs in Iran

Towards the Contributing Factors for the Inconsistency Between English as a Foreign Language (EFL) Teachers' Equity-Oriented Cognition and Practices

Students’ views on fairness in education: the importance of relational justice and stakes fairness

Teacher Cognition and Practice of Educational Equity in English as a Foreign Language Teaching

Fairer for all: Teachers identifying and solving problems with the assessment of English language learners in the mainstream classroom

Conceptions of Assessment Among Pakistani Teachers of English: Implications for Policy and Professional Development

Assessment Literacy: Secondary School Teachers’ Conception of Formative Assessment

Students’ perceptions of student voice in assessment within the context of Iran: the dynamics of culture, power relations, and student knowledge

FairAIED: Navigating Fairness, Bias, and Ethics in Educational AI Applications

The Fairness Connotation of Individualized Instruction and Paths of Implementation

Grading by AI makes me feel fairer? How different evaluators affect college students' perception of fairness

EFL teachers’ perceptions and practices of their language assessment knowledge

Classroom justice: Perceptions of fairness as a predictor of student motivation, learning, and aggression

Teachers' Perceptions of ELL Students: Do Their Attitudes Shape Their Instruction?