Abstract:Little is known whether peer reviewers use the same evaluation criteria and how they integrate the criteria into their overall judgment. This study therefore proposed two assessment styles based on theoretical perspectives and normative positions. According to the case-by-case style, referees use many and different criteria, weight criteria on a case-by-case basis, and integrate criteria in a complex, non-mechanical way into their overall judgment. According to the uniform style, referees use a small fraction of the available criteria, apply the same criteria, weight the criteria in the same way, and integrate the criteria based on simple rules (i.e., fast-and-frugal heuristics). These two styles were examined using a unique dataset from a career funding scheme that contained a comparatively large number of evaluation criteria. A heuristic (fast-and-frugal trees) and a complex procedure (logistic regression) were employed to describe how referees integrate the criteria into their overall judgment. The logistic regression predicted the referees' overall assessment with high accuracy and slightly more accurately than the fast-and-frugal trees. Overall, the results of this study support the uniform style but also indicate that the uniform style needs to be revised as follows: referees use many criteria and integrate the criteria using complex rules. However, and most importantly, the revised style could describe most - but not all - of the referees' judgments. Future studies should therefore examine how referees' judgments can be characterized in those cases where the uniform style failed. Moreover, the evaluation process of referees should be studied in more empirical and theoretical detail.

Are terminology planning evaluation and language policy and planning evaluation applicable to the evaluation of standardisation?

Mapping Evaluation, Appraisal and Stance in Discourse (2000–2015): A Bibliometric Analysis

Terminology and Language Planning

Purposes and criteria in network governance evaluation: How far does standard evaluation vocabulary takes us?

Normalisation de terminologies multilingues pour les TICE : techniques et enjeux

What evaluation criteria are used in policy evaluation research: A cross-field literature review

The Role of the Communicative Approach in the Development of Terminology

The evaluation of foreign-language-teacher education programmes

An Ecological View of Language Planning——A Review of "Language Planning: from Practice to Theory"

Evaluation in Urban Planning: Advances and Prospects

Quantitative analysis at local and national level: getting it right and why it matters

Quantitative evaluation approach for English–Catalan translation of soundscape perceptual attributes

How do referees integrate evaluation criteria into their overall judgment? Evidence from grant peer review

An enhanced terminology development and management approach for South African languages

Evaluating Environmental Education Programmes: Some issues and problems

Needs analysis in language teaching

The Communicative Theory of Terminology, A Linguistic Approach of Terms

Post Turing: Mapping the landscape of LLM Evaluation

Should Terminology Principles be re-examined?

A holistic model for evaluating the impact of individual technology-enhanced learning resources

Integrative evaluation: an emerging role for classroom studies of CAL