Correcting Selection Bias in Standardized Test Scores Comparisons

Onil Boussim

DOI: https://doi.org/10.48550/arXiv.2309.10642

2024-06-29

Abstract:This paper addresses the issue of sample selection bias when comparing countries using International assessments like PISA (Program for International Student Assessment). Despite its widespread use, PISA rankings may be biased due to different attrition patterns in different countries, leading to inaccurate comparisons. This study proposes a methodology to correct for sample selection bias using a quantile selection model. Applying the method to PISA 2018 data, I find that correcting for selection bias significantly changes the rankings (based on the mean) of countries' educational performances. My results highlight the importance of accounting for sample selection bias in international educational comparisons.

Econometrics,Applications

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the inaccuracy in ranking and comparison caused by sample selection bias when international student assessment projects (such as PISA) compare the educational performance of different countries. Specifically:

1. **Sample selection bias problem**: The coverage of international assessment projects such as PISA is not 100% of all target populations, and the coverage rate in some countries is relatively low. This may lead to only those students who are more likely to stay in the school system until the assessment age being included in the sample, making the performance of these countries seem better than it actually is.

2. **Impact on educational policies**: Comparisons between countries based on biased data may mislead educational policy - makers and affect the direction and decision - making of educational reforms. Therefore, correcting this sample selection bias is crucial for ensuring fair and accurate international educational comparisons.

To solve this problem, the author proposes a method based on the quantile selection model to correct the sample selection bias and applies it to the PISA data in 2018. The research results show that there are significant differences between the corrected rankings and the original rankings, emphasizing the importance of considering sample selection bias in international educational comparisons.

### Key point summary:
- **Problem background**: The sample selection bias in international student assessment projects (such as PISA) leads to unfair comparisons of educational performance between countries.
- **Solution**: A method based on the quantile selection model is proposed to correct the sample selection bias.
- **Application result**: By applying it to the PISA data in 2018, it is found that there are significant differences between the corrected rankings and the original rankings, which proves the existence of sample selection bias and its impact on rankings.

### Formula representation:
To better understand this method, here are some key formulas:

1. **Quantile representation of potential assessment scores**:
\[
Y^* = q_{Y^*}(U)
\]
where \( U \sim U[0, 1] \) is the rank of \( Y^* \).

2. **Selection - corrected quantile estimate**:
\[
q_{Y^*}(u) = q_{Y|S = 1}(\tilde{u})
\]
where \( \tilde{u}=P(U\leq u|S = 1) \) is the selection - corrected rank.

3. **Boundaries of partial identification**:
\[
q_{Y|S = 1}\left(\frac{\max\{u + p - 1,0\}}{p}\right)\leq q_{Y^*}(u)\leq q_{Y|S = 1}\left(\frac{\min\{u,p\}}{p}\right)
\]

4. **Point identification under parametric assumptions**:
\[
q_{Y^*}(u)=q_{Y|S = 1}\left(\frac{1}{p}\left(uF_{V,\theta_0}(u)-\int_0^u vdF_{V,\theta_0}(v)\right)\right)
\]
where \( \theta_0=\frac{1}{1 - p}-1 \).

Through these formulas, the author provides a practical and simple method to correct the sample selection problem, thereby improving the accuracy of international educational comparisons.