Abstract:Recent developments in the philosophy of validity, highlighting the importance of investigating the consequences of assessment use, provide theoretical support for the move toward performance assessment. The problem for validity researchers is finding the appropriate set of criteria and standards to simultaneously support the validity of an assessment-based interpretation and the validity of its impact on the educational system. My intent is to provide an integrative and critical review of the guidance available for conducting validity inquiry in the context of performance assessment. In the first section is a summary of the emerging consensus among measurement scholars—not reflected in the 1985 Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME])—about the centrality of construct validity to the evaluation of any assessment-based interpretation and about the importance of expanding the concept of validity to include explicit consideration of the consequences of assessment use. The description of this emerging consensus suggests general epistemological principles for validity inquiry. In the second section is a description and synthesis of various categories of questions, evidence, or criteria that have been used to describe or guide validity inquiry, either for assessment in general or for performance assessment in particular. These analytic schemes, like the traditional construct-content-criterion categories, highlight specific issues that their authors consider important for validity researchers to address. Each balances technical concerns about such issues as reliability, generalizability, and comparability with concerns about the consequences of assessment. In the final section is an overview of concerns, expressed largely by interpretive researchers, about validity criteria that privilege standardized forms of assessment, whether performance-based or multiple-choice. These arguments suggest the importance of further expanding the conception of validity inquiry to treat as problematic the epistemological principles used to warrant validity conclusions.

Improving Balance in Educational Measurement: A Legacy of E. F. Lindquist

Improving measurement-invariance assessments: correcting entrenched testing deficiencies

THE PSYCHOLOGY OF EDUCATIONAL MEASUREMENT

A validity framework for accountability: educational measurement and language testing

Measuring individual differences in statistical learning: Current pitfalls and possible solutions

Finding the missing Einsteins: Expanding the breadth of cognitive and noncognitive measures used in academic services

Achievement motivation theory: Balancing precision and utility

The use of test scores from large-scale assessment surveys: psychometric and statistical considerations

Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment

Exploring a New Application of Construct Specification Equations (CSEs) and Entropy: A Pilot Study with Balance Measurements

Getting at Student Understanding - the Key to Teachers’ Use of Test Data

Practical Significance of Item Misfit in Educational Assessments

From alpha to omega and beyond! A look at the past, present, and (possible) future of psychometric soundness in the Journal of Applied Psychology.

Understanding the State of the Art for Measurement in Chemistry Education Research: Examining the Psychometric Evidence

Estimating Test-Retest Reliability in the Presence of Self-Selection Bias and Learning/Practice Effects

Assessing Efficiency in Education: The Imperative of Curriculum-Based Measurements

A Multi-Faceted Approach to Scrutinizing the Reliability of a Measure of STEM Teacher Strategic Knowledge

From Donders’ Dilemma to Objective Internal Assessment: How Experimental Developmental Psychology Can Contribute to Moral Education

Reliability and competitiveness of a novel balance field test: A cross-sectional pilot study

Educational assessment without numbers

Looking for a broader mindset in psychometrics: the case for more participatory measurement practices