Abstract:Recent developments in the philosophy of validity, highlighting the importance of investigating the consequences of assessment use, provide theoretical support for the move toward performance assessment. The problem for validity researchers is finding the appropriate set of criteria and standards to simultaneously support the validity of an assessment-based interpretation and the validity of its impact on the educational system. My intent is to provide an integrative and critical review of the guidance available for conducting validity inquiry in the context of performance assessment. In the first section is a summary of the emerging consensus among measurement scholars—not reflected in the 1985 Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME])—about the centrality of construct validity to the evaluation of any assessment-based interpretation and about the importance of expanding the concept of validity to include explicit consideration of the consequences of assessment use. The description of this emerging consensus suggests general epistemological principles for validity inquiry. In the second section is a description and synthesis of various categories of questions, evidence, or criteria that have been used to describe or guide validity inquiry, either for assessment in general or for performance assessment in particular. These analytic schemes, like the traditional construct-content-criterion categories, highlight specific issues that their authors consider important for validity researchers to address. Each balances technical concerns about such issues as reliability, generalizability, and comparability with concerns about the consequences of assessment. In the final section is an overview of concerns, expressed largely by interpretive researchers, about validity criteria that privilege standardized forms of assessment, whether performance-based or multiple-choice. These arguments suggest the importance of further expanding the conception of validity inquiry to treat as problematic the epistemological principles used to warrant validity conclusions.

How not to evaluate a psychological measure: Rebuttal to criticism of the Defining Issues Test of moral judgment development by Curzer and colleagues

From Donders’ Dilemma to Objective Internal Assessment: How Experimental Developmental Psychology Can Contribute to Moral Education

Moral Judgement Development in Higher Education: Insights from the Defining Issues Test

Does Moral Education Improve Moral Judgment? A Meta-Analysis of Intervention Studies Using the Defining Issues Test

The Defining Issues Test and the Four Component Model: Contributions to professional education

Validity study using factor analyses on the Defining Issues Test-2 in undergraduate populations

Stages of moral judgment development: Applying item response theory to Defining Issues Test data

How to Assess Whether an Instructional Intervention Has an Effect on Learning

Attitude theory and measurement in implementation science: a secondary review of empirical studies and opportunities for advancement

Moral rationality and intuition: An exploration of relationships between the Defining Issues Test and the Moral Foundations Questionnaire

It looks like construct validity, but look again: Comment on Clutterbuck et al. (2021) and recommendations for test developers in the broad "empathy" domain

Alchemy and beyond: Indexing the Defining Issues Test.

Measuring moral reasoning using moral dilemmas: evaluating reliability, validity, and differential item functioning of the behavioural defining issues test (bDIT)

Contemporary Test Validity in Theory and Practice: A Primer for Discipline-Based Education Researchers

Challenges in Process Dissociation Measures for Moral Cognition

What is wrong with individual differences research?

Theoretical and methodological issues with testing the SCCT and RIASEC models: Comment on Lent, Sheu, and Brown (2010) and Lubinski (2010).

When better is the enemy of good: two cautionary tales of conceptual validity versus parsimony in clinical psychometric research

Evaluating the Measurement Reliabilities and Dimensionality of Developmental Idealism Measures

On the Proper Use of Statistical Analyses; a Comment on "Evaluation of Colorado Learning Attitudes about Science Survey" by Douglas et al

Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment