Abstract:Background: Public health interventions aim to reduce the burden of chronic non-communicable diseases. Implementing evidence-based interventions that are proven to be successful and effective is widely recognized as the best approach to addressing public health challenges. To avoid the development and implementation of less effective or successful or even harmful practices, clear criteria for the assessment of practices, that consider different dimensions of the interventions in public health, are needed. The main aim of the research was to test our Criteria and assessment procedure for recognizing good practices in the field of public health by estimating the consistency between the evaluators and thereby gaining insight into the adequacy and reliability of the criteria as well as to check how the evaluators understand the criteria and methodology and if it is properly used in assessing the interventions. Methods: The assessment of the interventions took place from 2021 to 2022. The individual evaluator's scores on the scale from 1 to 5 for each specific sub-criterion were collected, which was followed by a panel discussion to reach a final score for each sub-criterion. The inter-rater agreement was measured using percent overall agreement and Fleiss' kappa coefficient. Results: We found moderate inter-rater agreement on the level of the assessment criteria group. The lowest agreement was observed for the effectiveness and efficiency sub-criteria group, which also received the lowest scores from the evaluators. Challenges identified with the scoring process were due to the descriptive 1 to 5 scale and the varying specificity of the criteria. Conclusion: The results showed that studying consistency between evaluators can highlight areas for improvement or adjustment in the assessment criteria and enhance the quality of the assessment instrument. Therefore, such analysis would be useful part of both newly and well-established health promotion and prevention program registries.

Estimating Critical Values for Strength of Alignment Among Curriculum, Assessments, and Instruction

Evaluating the alignment between the Grade 9 mathematics Annual National Assessment and the TIMSS test items

Alignment between biology curriculum standards and five textbook editions: a content analysis

Systematic Characterization of the Effectiveness of Alignment in Large Language Models for Categorical Decisions

Alignment of the Intended, Planned, and Enacted Curriculum in General and Special Education and Its Relation to Student Achievement

EXAMINING THE CONTENT ALIGNMENT BETWEEN LANGUAGE CURRICULUM AND A LANGUAGE TEST IN CHINA

Choosing Between Cronbach's Coefficient Alpha, McDonald's Coefficient Omega, and Coefficient H: Confidence Intervals and the Advantages and Drawbacks of Interpretive Guidelines

An Analysis of Three States' Alignment between Language Arts and Mathematics Standards and Alternate Assessments

Evaluator's alignment as an important indicator of adequacy of the criteria and assessment procedure for recognizing the good practice in public health

One size does not fit all: a concurrent analysis of the framework for teaching and the Classroom Strategies Assessment System

The Lower-Order Expectations of High-Stakes Tests: A Four-State Analysis of Social Studies Standards and Test Alignment

The Impact of a Virtual Coaching Program to Improve Instructional Alignment to State Standards

A Multi-Faceted Approach to Scrutinizing the Reliability of a Measure of STEM Teacher Strategic Knowledge

SEAL: Systematic Error Analysis for Value ALignment

The stability of kindergarten teachers’ effectiveness: A generalizability study comparing the Framework For Teaching and the Classroom Assessment Scoring System

Understanding and Measuring Coach–Teacher Alliance: A Glimpse Inside the ‘Black Box’

THE ALIGNMENT BETWEEN THIRD-GRADE PRIMARY SCHOOL SCIENCE TEXTBOOKS AND CURRICULUM STANDARDS IN CHINA AND JAPAN

A validity framework for accountability: educational measurement and language testing

Alpha, Omega, and H Internal Consistency Reliability Estimates: Reviewing These Options and When to Use Them

Liberal-Conservative Hierarchies of Intercoder Reliability Estimators