Investigating the Reliability of CET6 Essay Scoring:An Application of Generalizability Theory and Many-facet Rasch Model

LI Hang
DOI: https://doi.org/10.13458/j.cnki.flatt.000423
2011-01-01
Abstract:Reliability is a necessary prerequisite for any test score to be valid.In order to evaluate the reliability of CET-6 essay scoring,analysis of the scoring of 100 CET-6 essays by 10 CET-6 raters was carried out using both Generalizability Theory and Many-facet Rasch Model.The results of Generalizability Theory analysis showed that the variance components for raters and residual(rater-test taker interaction confounded with random error) did account for a certain level of inconsistency in essay rating.Meanwhile,Many-facet Rasch Model analysis revealed that raters did show much variation in their severity.Furthermore,the bias analysis for rater-test taker interaction showed that there was a tendency for raters to rate higher ability writers more severely and lower ability writers more leniently than expected.The study also demonstrated that Generalizability Theory and Many-facet Rasch Model complement each other well,with the former offering group-level and the latter individual-level reliability information.
What problem does this paper attempt to address?