A Potential Outcomes View of Value-Added Assessment in Education

DB Rubin,EA Stuart,EL Zanutto

DOI: https://doi.org/10.3102/10769986029001103

2004-01-01

Journal of Educational and Behavioral Statistics

Abstract:Let’s See More Empirical Studies on Value-Added Modeling of Teacher Effects: A Reply to Raudenbush, Rubin, Stuart and Zanutto, and Reckase

What problem does this paper attempt to address?

Value-Added Modeling and Educational Accountability

Kimberlee C. Everson

DOI: https://doi.org/10.3102/0034654316637199

2016-07-09

Review of Educational Research

Abstract:Value-added estimates of teacher or school quality are increasingly used for both high- and low-stakes accountability purposes, making understanding of their limitations critical. A review of the recent value-added literature suggests three concerns with the state of the research. First, the issues receiving the most research attention have not always been the concerns of greatest importance to theorists or critics. Second, there has been insufficient research on the interactions among various issues or assumption violations. Third, some of the big issues in value-added modeling have been challenging to address and may require educators to step back and answer some underlying philosophical questions about the nature of teacher and school quality.

education & educational research
Teacher-Level Value-Added Models on Trial: Empirical and Pragmatic Issues of Concern Across Five Court Cases

Audrey Amrein-Beardsley,Kevin Close

DOI: https://doi.org/10.1177/0895904819843593

2019-04-21

Educational Policy

Abstract:Ongoing or recently completed across the United States are a series of lawsuits via which teacher plaintiffs are contesting how they are being evaluated using value-added models (VAMs) as part of states’/districts’ teacher accountability systems. To investigate the empirical and pragmatic matters addressed in court, researchers conducted a case study analysis of the documents submitted for five such cases. Researchers framed analyses using measurement concepts resident within the Standards for Educational and Psychological Testing, given issues with (a) reliability, (b) validity, (c) bias, (d) transparency, and (e) fairness, with emphases also on (f) whether VAMs are being used to make consequential decisions using concrete (e.g., not arbitrary) evidence, and (g) whether VAMs’ unintended consequences are also of legal pertinence and concern.

education & educational research
Teacher Value-Added at the High-School Level

Dan D. Goldhaber,Pete Goldschmidt,Fannie Tseng

DOI: https://doi.org/10.3102/0162373712466938

2013-06-01

Educational Evaluation and Policy Analysis

Abstract:This article reports on findings based on analyses of a unique dataset collected by ACT that includes information on student achievement in a variety of subjects at the high-school level. The authors examine the relationship between teacher effect estimates derived from value-added model (VAM) specifications employing different student learning assumptions. Specifically, they compare teacher effectiveness estimates derived from a traditional lagged score VAM using pretests and posttests in a single subject area to those derived from VAM specifications employing a cross-subject student fixed-effects approach. The latter approach offers advantages for teacher evaluation systems due to significantly reduced data requirements; however, there is evidence that both the estimated effect size of teacher quality as well as estimates of individual teacher performance vary depending on the VAM methodology. In particular, teacher effects identified based on within-student cross-subject variation results in significantly smaller effect size estimates than do those generated from a more traditional lagged score model. The correlation across model specification ranges from .25 to .96 depending on the subject area.

education & educational research
School value-added models for multivariate academic and non-academic outcomes: A more rounded approach to using student data to inform school accountability

Lucy Prior,Harvey Goldstein,George Leckie

DOI: https://doi.org/10.48550/arXiv.2001.01996

2020-01-07

Applications

Abstract:Education systems around the world increasingly rely on school value-added models to hold schools to account. These models typically focus on a limited number of academic outcomes, failing to recognise the broader range of non-academic student outcomes, attitudes and behaviours to which schools contribute. We explore how the traditional multilevel modelling approach to school value-added models can be extended to simultaneously analyse multiple academic and non-academic outcomes and thereby can potentially provide a more rounded approach to using student data to inform school accountability. We jointly model student attainment, absence and exclusion data for schools in England. We find different results across the three outcomes, in terms of the size and consistency of school effects, and the importance of adjusting for student and school characteristics. The results suggest the three outcomes are capturing fundamentally distinct aspects of school performance, recommending the consideration of non-academic outcomes in systems of school accountability.
Value-Added Models for Teacher Preparation Programs: Validity and Reliability Threats, and a Manageable Alternative

Michael P. Brady,Lawrence A. Heiser,Jazarae K. McCormick,James Forgan

DOI: https://doi.org/10.1080/00131725.2016.1173150

2016-05-23

The Educational Forum

Abstract:High-stakes standardized student assessments are increasingly used in value-added evaluation models to connect teacher performance to P–12 student learning. These assessments are also being used to evaluate teacher preparation programs, despite validity and reliability threats. A more rational model linking student performance to candidates who actually teach these students is presented. Preliminary findings with three candidate cohorts indicate that the majority of their students met learning objectives and showed substantial pre-to-post learning gains.
Stability of Teacher Value-Added Rankings Across Measurement Model and Scaling Conditions

Leslie R. Hawley,James A. Bovaird,ChaoRong Wu

DOI: https://doi.org/10.1080/08957347.2017.1316273

2017-04-14

Applied Measurement in Education

Abstract:Value-added assessment methods have been criticized by researchers and policy makers for a number of reasons. One issue includes the sensitivity of model results across different outcome measures. This study examined the utility of incorporating multivariate latent variable approaches within a traditional value-added framework. We evaluated the stability of teacher rankings across univariate and multivariate measurement model structures and scaling metric combinations using a cumulative cross-classified mixed effect model. Our results showed multivariate models were more stable across modeling conditions than univariate approaches. These findings suggest there is potential utility in incorporating multiple measures with teacher evaluation systems, yet future research will need to evaluate the degree to which models recover known population parameters via Monte Carlo simulation.

education & educational research,psychology, educational, mathematical
A Correlated Random Effects Model for Nonignorable Missing Data in Value-Added Assessment of Teacher Effects

Andrew T. Karl,Yan Yang,Sharon L. Lohr

DOI: https://doi.org/10.48550/arXiv.1710.01894

2017-10-05

Methodology

Abstract:Value-added models have been widely used to assess the contributions of individual teachers and schools to students' academic growth based on longitudinal student achievement outcomes. There is concern, however, that ignoring the presence of missing values, which are common in longitudinal studies, can bias teachers' value-added scores. In this article, a flexible correlated random effects model is developed that jointly models the student responses and the student missing data indicators. Both the student responses and the missing data mechanism depend on latent teacher effects as well as latent student effects, and the correlation between the sets of random effects adjusts teachers' value-added scores for informative missing data. The methods are illustrated with data from calculus classes at a large public university and with data from an elementary school district.
Value-Added Model (VAM) Scholars on Using VAMs for Teacher Evaluation After the Passage of the Every Student Succeeds Act

Matthew Ryan Lavery,Audrey Amrein-Beardsley,Tray Geiger,Margarita Pivovarova

DOI: https://doi.org/10.1177/016146812012200703

2020-07-01

Abstract:Background/Context The Race to the Top federal initiatives and requirements surrounding waivers of No Child Left Behind promoted expanded use of value-added models (VAMs) to evaluate teachers. Even after passage of the Every Student Succeeds Act (ESSA) relaxed these requirements, allowing more flexibility and local control, many states and districts continue to use VAMs in teacher evaluation systems, suggesting that they consider VAMs a valid measure of teacher effectiveness. Scholars in the fields of economics, education, and quantitative methods continue to debate several aspects of VAMs’ validity for this purpose, however. Purpose The purpose of this study was to directly ask the most experienced VAM scholars about validity of VAM use in teacher evaluation based on the aspects of validity described in the Standards for Educational and Psychological Testing and found in a review of high-quality peer-reviewed literature on VAMs. Participants We invited the 115 scholars listed as an author or coauthor of one or more of the 145 articles published on evaluating teachers with VAMs that have been published in prominent peer-reviewed journals between 2002 and implementation of ESSA in 2016. In this article, we analyze data from 36 respondents (12 economists, 13 educators, and 11 methodologists) who rated themselves as “experienced scholars,” “experts,” or “leading experts” on VAMs. Research Design This article reports both quantitative and qualitative analyses of a survey questionnaire completed by experienced VAM scholars. Findings Analyses of 44 Likert-scale items indicate that respondents were generally neutral or mixed toward the use of VAMs in teacher evaluation, though responses from educational researchers were more critical of VAM use than were responses from economists and quantitative methodologists. Qualitative analysis of free response comments suggests that participants oppose exclusive or high-stakes use of VAMs but are more supportive of their use as a component of evaluation systems that use multiple measures. Conclusions These findings suggest that scholars and stakeholders from different disciplines and backgrounds think about VAMs and VAM use differently. We argue that it is important to understand and address stakeholders’ multiple perspectives to find the common ground on which to build consensus.
Missing data in value-added modeling of teacher effects

Daniel F. McCaffrey,J. R. Lockwood

DOI: https://doi.org/10.48550/arXiv.1108.2167

2011-08-10

Applications

Abstract:The increasing availability of longitudinal student achievement data has heightened interest among researchers, educators and policy makers in using these data to evaluate educational inputs, as well as for school and possibly teacher accountability. Researchers have developed elaborate "value-added models" of these longitudinal data to estimate the effects of educational inputs (e.g., teachers or schools) on student achievement while using prior achievement to adjust for nonrandom assignment of students to schools and classes. A challenge to such modeling efforts is the extensive numbers of students with incomplete records and the tendency for those students to be lower achieving. These conditions create the potential for results to be sensitive to violations of the assumption that data are missing at random, which is commonly used when estimating model parameters. The current study extends recent value-added modeling approaches for longitudinal student achievement data Lockwood et al. [J. Educ. Behav. Statist. 32 (2007) 125--150] to allow data to be missing not at random via random effects selection and pattern mixture models, and applies those methods to data from a large urban school district to estimate effects of elementary school mathematics teachers. We find that allowing the data to be missing not at random has little impact on estimated teacher effects. The robustness of estimated teacher effects to the missing data assumptions appears to result from both the relatively small impact of model specification on estimated student effects compared with the large variability in teacher effects and the downweighting of scores from students with incomplete data.
Models for Value-Added Modeling of Teacher Effects

Daniel F McCaffrey,J R Lockwood,Daniel Koretz,Thomas A Louis,Laura Hamilton

DOI: https://doi.org/10.3102/10769986029001067

Abstract:The use of complex value-added models that attempt to isolate the contributions of teachers or schools to student development is increasing. Several variations on these models are being applied in the research literature, and policy makers have expressed interest in using these models for evaluating teachers and schools. In this article, we present a general multivariate, longitudinal mixed-model that incorporates the complex grouping structures inherent to longitudinal student data linked to teachers. We summarize the principal existing modeling approaches, show how these approaches are special cases of the proposed model, and discuss possible extensions to model more complex data structures. We present simulation and analytical results that clarify the interplay between estimated teacher effects and repeated outcomes on students over time. We also explore the potential impact of model misspecifications, including missing student covariates and assumptions about the accumulation of teacher effects over time, on key inferences made from the models. We conclude that mixed models that account for student correlation over time are reasonably robust to such misspecifications when all the schools in the sample serve similar student populations. However, student characteristics are likely to confound estimated teacher effects when schools serve distinctly different populations.
An Evaluation of Empirical Bayes’s Estimation of Value-Added Teacher Performance Measures

Cassandra M. Guarino,Michelle Maxfield,Mark D. Reckase,Paul N. Thompson,Jeffrey M. Wooldridge

DOI: https://doi.org/10.3102/1076998615574771

2015-04-01

Journal of Educational and Behavioral Statistics

Abstract:Empirical Bayes’s (EB) estimation has become a popular procedure used to calculate teacher value added, often as a way to make imprecise estimates more reliable. In this article, we review the theory of EB estimation and use simulated and real student achievement data to study the ability of EB estimators to properly rank teachers. We compare the performance of EB estimators with that of other widely used value-added estimators under different teacher assignment scenarios. We find that, although EB estimators generally perform well under random assignment (RA) of teachers to classrooms, their performance suffers under nonrandom teacher assignment. Under non-RA, estimators that explicitly (if imperfectly) control for the teacher assignment mechanism perform the best out of all the estimators we examine. We also find that shrinking the estimates, as in EB estimation, does not itself substantially boost performance.

education & educational research,psychology, mathematical,social sciences, mathematical methods
Accounting for Co-Teaching: A Guide for Policymakers and Developers of Value-Added Models

Eric Isenberg,Elias Walsh

DOI: https://doi.org/10.1080/19345747.2014.974232

2015-01-02

Journal of Research on Educational Effectiveness

Abstract:We outline the options available to policymakers for addressing co-teaching in a value-added model. Building on earlier work, we propose an improvement to a method of accounting for co-teaching that treats co-teachers as teams, with each teacher receiving equal credit for co-taught students. Hock and Isenberg (2012) described a method known as the Full Roster Method (FRM) that is feasible and practical, but it effectively counts co-taught students more than once—these students receive a full weight with each of their teachers, so such students receive extra weight when calculating the relationship between student characteristics and achievement. The improvement, known as the Full Roster-Plus Method, allows co-taught students to receive full weight with their teachers, but all students contribute equally to the calculation of the relationship between student characteristics and achievement. To investigate how the application of this method empirically changes value-added estimates, we use data from District of Columbia Public Schools. We find that there are very small empirical differences between the two methods.

education & educational research
A Rational and Manageable Value-Added Model for Teacher Preparation Programs

Michael P. Brady,Katie Miller,Jazarae McCormick,Lawrence A. Heiser

DOI: https://doi.org/10.1177/0895904816673741

2016-10-21

Educational Policy

Abstract:Educators struggle with “value-added” teacher evaluation models based on high-stakes student assessments. Despite validity and reliability threats, these models evaluate university-based teacher preparation programs (TPPs), and play a role in state and professional accreditation. This study reports a more rational value-added evaluation model linking student performance to teacher candidates’ lessons during Practicum and Student Teaching. Results indicate that K-12 students showed learning gains on these lessons, with mixed findings on comparisons of part-time to full-time internships, academic and functional lessons, and candidates’ grade point averages (GPAs). Results indicated that teacher candidates’ lessons are a viable value-added model (VAM) alternative for TPPs.

education & educational research
A comparison of value-added models for school accountability

George Leckie,Lucy Prior

DOI: https://doi.org/10.1080/09243453.2022.2032763

2022-02-07

School Effectiveness and School Improvement

Abstract:School accountability systems increasingly hold schools to account for their performances using value-added models purporting to measure the effects of schools on student learning. The most common approach is to fit a linear regression of student current achievement on student prior achievement, where the school effects are the school means of the predicted residuals. In the literature, further adjustments are usually made for student sociodemographics and sometimes school composition and “non-malleable” characteristics. However, accountability systems typically make fewer adjustments: for transparency to end users, because data are unavailable or of insufficient quality, or for ideological reasons. There is therefore considerable interest in understanding the extent to which simpler models give similar school effects to more theoretically justified but complex models. We explore these issues via a case study and empirical analysis of England’s “Progress 8” secondary school accountability system.

education & educational research
An alternative, curriculum-based value-added model for teacher preparation programmes: a research summary

Michael P. Brady

DOI: https://doi.org/10.1080/00131911.2019.1601615

2019-07-17

Educational Review

Abstract:Teacher preparation programmes (TPPs) find many ways to shape the development of their teacher candidates while bridging the research to practice gap in university and school partnerships. Recently, accountability models built on high-stakes standardised student assessments (Value-Added Models [VAM]) have been mandated as a way to evaluate the effectiveness of teacher and teacher preparation efforts. Adopting a metric with known validity and reliability flaws has unintended consequences that affect teachers, TPPs, and K-12 students. This article examines the current climate surrounding teacher and TPP evaluation, and presents the unintended consequences of using a flawed metric for VAM efforts. The article provides a rationale for an alternative, curriculum-based VAM for TPPs, and summarises a series of four investigations that explored this model and serve as a rationale for further implementation by other TPPs.

education & educational research
Accounting for Student Disadvantage in Value-Added Models

Eric Parsons,Cory Koedel,Li Tan

DOI: https://doi.org/10.3102/1076998618803889

2018-10-11

Journal of Educational and Behavioral Statistics

Abstract:We study the relative performance of two policy-relevant value-added models—a one-step fixed effect model and a two-step aggregated residuals model—using a simulated data set well grounded in the value-added literature. A key feature of our data generating process is that student achievement depends on a continuous measure of economic disadvantage. This is a realistic condition that has implications for model performance because researchers typically have access to only a noisy, binary measure of disadvantage. We find that one- and two-step value-added models perform similarly across a wide range of student and teacher sorting conditions, with the two-step model modestly outperforming the one-step model in conditions that best match observed sorting in real data. A reason for the generally superior performance of the two-step model is that it better handles the use of an error-prone, dichotomous proxy for student disadvantage.

education & educational research,psychology, mathematical,social sciences, mathematical methods
The Effect of the Prior Teacher on Value-Added

Michael Gilraine,Odhrain McCarthy

DOI: https://doi.org/10.1162/rest_a_01409

2024-02-09

Abstract:Abstract We show that teachers' value-added (‘VA’) depend on the VA of the teachers that preceded them. To do so, we use administrative data from North Carolina and find that a one-standard deviation increase in last grade's mean teacher VA causes a 0.08σ decrease in teacher VA. Controlling for prior teacher assignment eliminates this bias. Under a benchmark policy that releases teachers in the bottom five percent of the VA distribution, thirty-two percent of teachers are incorrectly released using traditional techniques. Our results highlight the importance of incorporating dynamic features of education production into the estimation of teacher quality.

economics,social sciences, mathematical methods
Rethinking principal effects on student outcomes

Brendan Bartanen,Aliza N. Husain,David D. Liebowitz

DOI: https://doi.org/10.1016/j.jpubeco.2024.105115

IF: 8.262

2024-04-11

Journal of Public Economics

Abstract:School principals are viewed as critical actors to improve student outcomes, but there remain important methodological questions about how to measure principals' effects. We propose a framework for measuring principals' contributions to student outcomes and apply it empirically using data from Tennessee, New York City, and Oregon. As commonly implemented, value-added models misattribute to principals changes in student performance caused by unobserved time-varying factors over which principals exert minimal control, leading to biased estimates of individual principals' effectiveness and an overstatement of the magnitude of principal effects. Based on our framework, which better accounts for bias from time-varying factors, we find that little of the variation in student test scores or attendance is explained by persistent effectiveness differences between principals. Across contexts, the estimated standard deviation of principal value-added is roughly 0.03 student-level standard deviations in math achievement and 0.01 standard deviations in reading.

economics
The Sensitivity of Value-Added Modeling to the Creation of a Vertical Score Scale

Derek C. Briggs,Jonathan P. Weeks

DOI: https://doi.org/10.1162/edfp.2009.4.4.384

2009-10-01

Education Finance and Policy

Abstract:The purpose of this study was to evaluate the sensitivity of growth and value-added modeling to the way an underlying vertical score scale has been created. Longitudinal item-level data were analyzed with both student- and school-level identifiers for the entire state of Colorado between 2003 and 2006. Eight different vertical scales were established on the basis of choices made for three key variables: the item response theory modeling approach, the calibration approach, and the student proficiency estimation approach. Each scale represented a methodological approach that was psychometrically defensible. Longitudinal values from each scale were used as the outcome in a commonly used value-added model (the “layered model” popularized by William Sanders) as a means of estimating school effects. Our findings suggest that while the ordering of estimating school effects is insensitive to the underlying vertical scale, the precision of such value-added estimates can be quite sensitive to the combinations of choices made in the creation of the scale.

education & educational research,economics
Uncertainty in Rank Estimation: Implications for Value-Added Modeling Accountability Systems

J R Lockwood,Thomas A Louis,Daniel F McCaffrey

DOI: https://doi.org/10.3102/10769986027003255

2002-01-01

Abstract:Accountability for public education often requires estimating and ranking the quality of individual teachers or schools on the basis of student test scores. Although the properties of estimators of teacher-or-school effects are well established, less is known about the properties of rank estimators. We investigate performance of rank (percentile) estimators in a basic, two-stage hierarchical model capturing the essential features of the more complicated models that are commonly used to estimate effects. We use simulation to study mean squared error (MSE) performance of percentile estimates and to find the operating characteristics of decision rules based on estimated percentiles. Each depends on the signal-to-noise ratio (the ratio of the teacher or school variance component to the variance of the direct, teacher- or school-specific estimator) and only moderately on the number of teachers or schools. Results show that even when using optimal procedures, MSE is large for the commonly encountered variance ratios, with an unrealistically large ratio required for ideal performance. Percentile-specific MSE results reveal interesting interactions between variance ratios and estimators, especially for extreme percentiles, which are of considerable practical import. These interactions are apparent in the performance of decision rules for the identification of extreme percentiles, underscoring the statistical and practical complexity of the multiple-goal inferences faced in value-added modeling. Our results highlight the need to assess whether even optimal percentile estimators perform sufficiently well to be used in evaluating teachers or schools.

A Potential Outcomes View of Value-Added Assessment in Education

Value-Added Modeling and Educational Accountability

Teacher-Level Value-Added Models on Trial: Empirical and Pragmatic Issues of Concern Across Five Court Cases

Teacher Value-Added at the High-School Level

School value-added models for multivariate academic and non-academic outcomes: A more rounded approach to using student data to inform school accountability

Value-Added Models for Teacher Preparation Programs: Validity and Reliability Threats, and a Manageable Alternative

Stability of Teacher Value-Added Rankings Across Measurement Model and Scaling Conditions

A Correlated Random Effects Model for Nonignorable Missing Data in Value-Added Assessment of Teacher Effects

Value-Added Model (VAM) Scholars on Using VAMs for Teacher Evaluation After the Passage of the Every Student Succeeds Act

Missing data in value-added modeling of teacher effects

Models for Value-Added Modeling of Teacher Effects

An Evaluation of Empirical Bayes’s Estimation of Value-Added Teacher Performance Measures

Accounting for Co-Teaching: A Guide for Policymakers and Developers of Value-Added Models

A Rational and Manageable Value-Added Model for Teacher Preparation Programs

A comparison of value-added models for school accountability

An alternative, curriculum-based value-added model for teacher preparation programmes: a research summary

Accounting for Student Disadvantage in Value-Added Models

The Effect of the Prior Teacher on Value-Added

Rethinking principal effects on student outcomes

The Sensitivity of Value-Added Modeling to the Creation of a Vertical Score Scale

Uncertainty in Rank Estimation: Implications for Value-Added Modeling Accountability Systems