Abstract:Abstract The present study attempted to to investigate factors which affect EFL writing scores through using generalizability theory (G-theory). To this purpose, one hundred and twenty students participated in one independent and one integrated writing tasks. Proceeding, their performances were scored by six raters: one self-rating, three peers,-rating and two instructors-rating. The main purpose of the sudy was to determine the relative and absolute contributions of different facets such as student, rater, task, method of scoring, and background of education to the validity of writing assessment scores. The results indicated three major sources of variance: (a) the student by task by method of scoring (nested in background of education) interaction (STM:B) with 31.8% contribution to the total variance, (b) the student by rater by task by method of scoring (nested in background of education) interaction (SRTM:B) with 26.5% of contribution to the total variance, and (c) the student by rater by method of scoring (nested in background of education) interaction (SRM:B) with 17.6% of the contribution. With regard to the G-coefficients in G-study (relative G-coefficient ≥ 0.86), it was also found that the result of the assessment was highly valid and reliable. The sources of error variance were detected as the student by rater (nested in background of education) (SR:B) and rater by background of education with 99.2% and 0.8% contribution to the error variance, respectively. Additionally, ten separate G-studies were conducted to investigate the contribution of different facets across rater, task, and methods of scoring as differentiation facet. These studies suggested that peer rating, analytical scoring method, and integrated writing tasks were the most reliable and generalizable designs of the writing assessments. Finally, five decision-making studies (D-studies) in optimization level were conducted and it was indicated that at least four raters (with G-coefficient = 0.80) are necessary for a valid and reliable assessment. Based on these results, to achieve the greatest gain in generalizability, teachers should have their students take two writing assessments and their performance should be rated on at least two scoring methods by at least four raters.

A Measure of Efl Argumentative Writing Cognitive Load: Scale Development and Validation

Analyzing Critical Thinking Elements in the Argument Structure of Non-English-Major Chinese Undergraduate Students' Writing

Assessing L2 integrated writing self-efficacy: Development and validation of a scale

Learning English as a Foreign Language Writing Skills in Collaborative Settings: A Cognitive Load Perspective

Constructing and validating a self-assessment scale for Chinese college English-major students’ feedback knowledge repertoire in EFL academic writing: Item response theory and factor analysis approaches

Predicting Chinese EFL Learners’ Human‐rated Writing Quality in Argumentative Writing Through Multidimensional Computational Indices of Lexical Complexity

Diagnosing Writing Ability Using China’s Standards of English Language Ability: Application of Cognitive Diagnosis Models

Development and Validation of a Diagnostic Rating Scale for EFL Writing in China

Development and Validation of a Theory-Based Questionnaire to Measure Different Types of Cognitive Load

A Comparison of EFL Raters' Essay-Rating Processes across Two Types of Rating Scales.

Syntactic Complexity Development in the Writings of EFL Learners: Insights from a Dependency Syntactically-Annotated Corpus

Understanding instructional design effects by differentiated measurement of intrinsic, extraneous, and germane cognitive load

Conceptualizing Writing Self‐Efficacy in English As a Foreign Language Contexts: Scale Validation Through Structural Equation Modeling

The affectability of writing assessment scores: a G-theory analysis of rater, task, and scoring method contribution

Exploring and modeling the reading-writing connection in EFL integrated writing

A Questionnaire-Based Validation Of Multidimensional Models Of Self-Regulated Learning Strategies

Validation of metacognitive academic writing strategies and the predictive effects on academic writing performance in a foreign language context

Measuring Cognitive Load: Are There More Valid Alternatives to Likert Rating Scales?

EXAMINING THE RELATIVE EFFECTS OF TASK COMPLEXITY AND COGNITIVE DEMANDS ON STUDENTS’ WRITING IN A SECOND LANGUAGE

The development and validation of an L2 student digital multimodal composing competence scale

Validating the Multi-Dimensional Structure of Self-Efficacy Beliefs in Peer Feedback for L2 Writing: A Bifactor-Exploratory Structural Equation Modeling Approach