Abstract:The precarious state of the educational system in the inner cities of the United States, as well as its potential causes and solutions, have been popular topics of debate in recent years. Part of the difficulty in resolving this debate is the lack of solid empirical evidence regarding the true impact of educational initiatives. The efficacy of so-called “school choice” programs has been a particularly contentious issue. A current multimillion dollar program, the School Choice Scholarship Foundation Program in New York, randomized the distribution of vouchers in an attempt to shed some light on this issue. This is an important time for school choice, because on June 27, 2002 the U.S. Supreme Court upheld the constitutionality of a voucher program in Cleveland that provides scholarships both to secular and religious private schools. Although this study benefits immensely from a randomized design, it suffers from complications common to such research with human subjects: noncompliance with assigned “treatments” and missing data. Recent work has revealed threats to valid estimates of experimental effects that exist in the presence of noncompliance and missing data, even when the goal is to estimate simple intention-to-treat effects. Our goal was to create a better solution when faced with both noncompliance and missing data. This article presents a model that accommodates these complications that is based on the general framework of “principal stratification” and thus relies on more plausible assumptions than standard methodology. Our analyses revealed positive effects on math scores for children who applied to the program from certain types of schools—those with average test scores below the citywide median. Among these children, the effects are stronger for children who applied in the first grade and for African-American children.

Discussion of "on State Education Statistics": A Difficulty with Regression Analyses of Regional Test Score Averages

Difficulties with Regression Analyses of Age-Adjusted Rates.

Regression with race-modifiers: towards equity and interpretability

Aggregating Student Achievement Trends Across States With Different Tests: Using Standardized Slopes as Effect Sizes

Estimating Returns to College Attainment: Comparing Survey and State Administrative Data-Based Estimates

Range of Meanings: A Sequential Mixed Methods Study of How English Language Learners Encounter Assessment Items on Descriptive Statistics

The use of test scores from large-scale assessment surveys: psychometric and statistical considerations

Principal Stratification Approach to Broken Randomized Experiments

Missing data in value-added modeling of teacher effects

Regression to the Mean's Impact on the Synthetic Control Method: Bias and Sensitivity Analysis

Implications of bias in automated writing quality scores for fair and equitable assessment decisions.

THE ACCEPTABILITY OF REGRESSION SOLUTIONS: ANOTHER LOOK AT COMPUTATIONAL ACCURACY

A Modified General Location Model for Noncompliance With Missing Data: Revisiting the New York City School Choice Scholarship Program Using Principal Stratification

On the Proper Use of Statistical Analyses; a Comment on "Evaluation of Colorado Learning Attitudes about Science Survey" by Douglas et al

Beta-binomial meta-analysis of individual differences based on sample means and standard deviations: Studying reliability of sum scores of binary items.

Why test? Talent loss and enrollment loss

Uncovering student errors in measures of dispersion: An APOS theory analysis in high school statistics education

COMPARING HIGH SCHOOLS WITH RESPECT TO STUDENT PERFORMANCE IN UNIVERSITY

Randomization-based confidence sets for the local average treatment effect

A Comparative Study of Critical Thinking Scores in Teacher Evaluations

Comparing standard deviation effects across contexts