Abstract:Background: Simulation plays a vital role in health professions assessment. This review provides a primer on assessment validation for educators and education researchers. We focus on simulation-based assessment of health professionals, but the principles apply broadly to other assessment approaches and topics. Key principles: Validation refers to the process of collecting validity evidence to evaluate the appropriateness of the interpretations, uses, and decisions based on assessment results. Contemporary frameworks view validity as a hypothesis, and validity evidence is collected to support or refute the validity hypothesis (i.e., that the proposed interpretations and decisions are defensible). In validation, the educator or researcher defines the proposed interpretations and decisions, identifies and prioritizes the most questionable assumptions in making these interpretations and decisions (the "interpretation-use argument"), empirically tests those assumptions using existing or newly-collected evidence, and then summarizes the evidence as a coherent "validity argument." A framework proposed by Messick identifies potential evidence sources: content, response process, internal structure, relationships with other variables, and consequences. Another framework proposed by Kane identifies key inferences in generating useful interpretations: scoring, generalization, extrapolation, and implications/decision. We propose an eight-step approach to validation that applies to either framework: Define the construct and proposed interpretation, make explicit the intended decision(s), define the interpretation-use argument and prioritize needed validity evidence, identify candidate instruments and/or create/adapt a new instrument, appraise existing evidence and collect new evidence as needed, keep track of practical issues, formulate the validity argument, and make a judgment: does the evidence support the intended use? Conclusions: Rigorous validation first prioritizes and then empirically evaluates key assumptions in the interpretation and use of assessment scores. Validation science would be improved by more explicit articulation and prioritization of the interpretation-use argument, greater use of formal validation frameworks, and more evidence informing the consequences and implications of assessment.

Validation Frameworks and Methods

Validation of a videoconferenced speaking test

Chapter 10 Validation of a Model Appropriateness Framework Using the Elbe Decision Support System

Development and Validation of A Computer Adaptive Efl Test

The New Understanding of Validity Concept

THE DEVELOPMENT OF VALIDITY AND VALIDATION IN LANGUAGE TESTING

A contemporary approach to validity arguments: a practical guide to Kane's framework

ValiText -- a unified validation framework for computational text-based measures of social constructs

Validation Obligations: A Novel Approach to Check Compliance between Requirements and their Formal Specification

Validation of educational assessments: a primer for simulation and beyond

Where Assessment Validation and Responsible AI Meet

An Extensive Knowledge Mapping Review of Measurement and Validity in Language Assessment and SLA Research

CheckEval: Robust Evaluation Framework using Large Language Model via Checklist

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

Configuration Validation with Large Language Models

Validation-Driven Development

A validity framework for accountability: educational measurement and language testing

A Validation and Quality Assessment Method with Metamorphic Relations for Unsupervised Machine Learning Software.

METTLE: A Metamorphic Testing Approach To Validating Unsupervised Machine Learning Methods

Validity: A Framework for Cross-Disciplinary Collaboration in Mining Indicators of Learning from Mooc Forums

LLM4VV: Exploring LLM-as-a-Judge for Validation and Verification Testsuites