Abstract:Abstract The present study attempted to to investigate factors which affect EFL writing scores through using generalizability theory (G-theory). To this purpose, one hundred and twenty students participated in one independent and one integrated writing tasks. Proceeding, their performances were scored by six raters: one self-rating, three peers,-rating and two instructors-rating. The main purpose of the sudy was to determine the relative and absolute contributions of different facets such as student, rater, task, method of scoring, and background of education to the validity of writing assessment scores. The results indicated three major sources of variance: (a) the student by task by method of scoring (nested in background of education) interaction (STM:B) with 31.8% contribution to the total variance, (b) the student by rater by task by method of scoring (nested in background of education) interaction (SRTM:B) with 26.5% of contribution to the total variance, and (c) the student by rater by method of scoring (nested in background of education) interaction (SRM:B) with 17.6% of the contribution. With regard to the G-coefficients in G-study (relative G-coefficient ≥ 0.86), it was also found that the result of the assessment was highly valid and reliable. The sources of error variance were detected as the student by rater (nested in background of education) (SR:B) and rater by background of education with 99.2% and 0.8% contribution to the error variance, respectively. Additionally, ten separate G-studies were conducted to investigate the contribution of different facets across rater, task, and methods of scoring as differentiation facet. These studies suggested that peer rating, analytical scoring method, and integrated writing tasks were the most reliable and generalizable designs of the writing assessments. Finally, five decision-making studies (D-studies) in optimization level were conducted and it was indicated that at least four raters (with G-coefficient = 0.80) are necessary for a valid and reliable assessment. Based on these results, to achieve the greatest gain in generalizability, teachers should have their students take two writing assessments and their performance should be rated on at least two scoring methods by at least four raters.

The Impact of Content and Mechanics on Judgments of Writing Quality

Analyzing Critical Thinking Elements in the Argument Structure of Non-English-Major Chinese Undergraduate Students' Writing

A Comparison of EFL Raters' Essay-Rating Processes across Two Types of Rating Scales.

Genre Effect on L2 Syntactic Complexity and Holistic Rating for Writing Quality of Intermediate EFL Learners

Organization of ideas in writing: what are raters sensitive to?

The Influence of EFL Proficiency on Written Products in Independent and Integrated Writing Tasks

The affectability of writing assessment scores: a G-theory analysis of rater, task, and scoring method contribution

Comparing computer-based and paper-based rating modes in an English writing test

Towards more valid scoring criteria for integrated reading-writing and listening-writing summary tasks

The impact of essay organization and overall quality on the holistic scoring of EFL writing: Perspectives from classroom english teachers and national writing raters

Comparative approaches to the assessment of writing: Reliability and validity of benchmark rating and comparative judgement

Rater cognitive processes in integrated writing tasks: from the perspective of problem-solving

The impact of writer nationality on mainstream teachers' judgments of composition quality

Assessing the Quality of Argumentative Texts: Examining the General Agreement Between Different Rating Procedures and Exploring Inferences of (Dis)agreement Cases

Investigating the Effect of Rater Training on Differential Rater Function in Assessing Academic Writing Skills of Higher Education Students

An analysis of teaching strategies designed to improve written communication skills

Enhancing Writing Proficiency: The Role of Model Essays as Corrective Feedback Tools in IELTS Writing Task Achievement and Coherence/Cohesion

The role and features of peer assessment feedback in college English writing

Toward a Transparent Construct of Reading-to-Write Tasks: The Interface Between Discourse Features and Proficiency

Writing Instruction Improves Students' Writing Skills Differentially Depending on Focal Instruction and Children: A Meta-Analysis for Primary Grade Students

Holistic and Analytic Assessments of the TOEFL iBT® Integrated Writing Task