Investigating the reliability of CET-SET using Multi-Facet Rasch Model

He Lianzhen,Zhang Jie

2008-01-01

Abstract:For a test to be valid, it must be reliable. In performance assessment, candidates’ test scores are the result of the interaction of many facets and therefore the reliability of performance assessment cannot be judged by inter-rater or intra-rater consistency alone. Based upon raw scores of a CET-SET at one of its testing centers, the present research investigates and models possible sources of score variance within the framework of Many-facet Rasch Model (MFRM). The results demonstrate statistically significant differences among all facets including rater severity, task difficulty, rating criteria, rating scale. MFRM, as an extension of the classical Rasch Model, helps to measure the effect of various facets on test scores in performance assessment and generates fair scores for each candidate, minimizing the variability due to facets other than candidates’ ability. MFRM manifests itself as an effective means for detecting whether each test method facet functions properly in such a performance assessment setting and for providing useful feedback for test improvement.