Abstract:Abstract It is widely believed that raters’ cognition is an important aspect of writing assessment, as it has both logical and temporal priority over scores. Based on a critical review of previous research in this area, it is found that raters’ cognition can be boiled to two fundamental issues: building text images and strategies for articulating scores. Compared to the scoring contexts of previous research, the TEM 8 integrated writing task scoring scale has unique features. It is urgent to know how raters build text images and how they articulate scores for text images in the specific context of rating TEM8 compositions. In order to answer these questions, the present study conducted qualitative research by considering raters as problem solvers in the light of problem-solving theory. Hence, 6 highly experienced raters were asked to verbalize their thoughts simultaneously while rating TEM 8 essays, supplemented by a retrospective interview. Analyzing the collected protocols, we found that with regard to research question 1, the raters went through two stages by setting building text images as isolated nodes and building holistic text images for each dimension as two sub-goals, respectively. In order to achieve the first sub-goal, raters used strategies such as single foci evaluating, diagnosing, and comparing; for the second sub-goal, they mainly used synthesizing and comparing. Regarding the second question, the results showed that they resorted to two groups of strategies: demarcating boundaries between scores within a dimension and discriminating between dimensions, each group consisting of more specific processes. Each of the extracted processes was defined clearly and their relationships were delineated, on the basis of which a new working model of the rating process was finalized. Overall, the present study deepens our understanding of rating processes and provides evidence for the scoring validity of the TEM 8 integrated writing test. It also provides implications for rating practice, such as the need for the distinction between two types of analytical rating scales.

Comparing computer-based and paper-based rating modes in an English writing test

Comparative Study about the Impacts of Chinese Automated Writing Evaluation (AWE) Feedback and Teacher Feedback on Middle School Students’ Writing Practice

A Comparison of EFL Raters' Essay-Rating Processes across Two Types of Rating Scales.

Examining mode effects for an adapted Chinese critical thinking assessment

Multi-faceted Rasch measurement and bias patterns in EFL writing performance assessment

Rater cognitive processes in integrated writing tasks: from the perspective of problem-solving

STUDY OF SOURCES OF SCORE VARIABILITY IN PERFORMANCE ASSESSMENT USING MFRM : A CASE OF SPEAKING TESTIN PETS BAND3 ?

Comparing Test-Taking Effort Between Paper-Based and Computer-Based Tests

Diagnosing Writing Ability Using China’s Standards of English Language Ability: Application of Cognitive Diagnosis Models

Scoring and Consequential Validity Evidence of Computer- and Paper-Based Writing Tests in Times of Change

Application of an Automated Essay Scoring engine to English writing assessment using Many-Facet Rasch Measurement

Comparative approaches to the assessment of writing: Reliability and validity of benchmark rating and comparative judgement

Towards more valid scoring criteria for integrated reading-writing and listening-writing summary tasks

A study of validity and reliability of Internet-based English writing testing

Establishing Comparability Across Writing Tasks With Picture Prompts of Three Alternate Tests

Online test administration results in students selecting more responses to multiple-choice-multiple-response items

Comparison of Automatic and Expert Teachers’ Rating of Computerized English Listening-Speaking Test

Validity of score interpretations on an online English placement writing test

An automated essay scoring systems: a systematic literature review

Applying Unidimensional and Multidimensional Item Response Theory Models in Testlet-Based Reading Assessment

From paper to pixels: A comparison of paper and computer formats in psychological assessment