Conducting Influential Impact Evaluations in China: the Experience of the Rural Education Action Project
Mathew Boswell,Scott Rozelle,Linxiu Zhang,Chengfang Liu,Renfu Luo,Yaojiang Shi
DOI: https://doi.org/10.23846/wp0010
2012-01-01
Abstract:Click to increase image sizeClick to decrease image size Notes 1. Although RCTs have many powerful features, there is a continuing debate among evaluators concerning the strengths and weaknesses of RCTs. Many evaluations have revealed some of the theoretical and practical limitations of RCT designs. Some of the limitations that have been mentioned include: the designs are rigid and inflexible and do not adapt well to changes in project design or implementation (which occur frequently) or to the changes in context in which the project is implemented; mono-method bias (relying on a single data collection method and outcome indicator; construct validity (simplifying complex outcome indicators to a single indicator that is easier to manipulate); difficulties in measuring qualitative dimensions; lack of attention to contextual variables (economic, political, institutional, socio-cultural; ignoring the process of project implementation; and long delays in providing feedback to policy-makers (compared with other evaluation models that are designed to provide feedback to managers and policy-makers during implementation). There are ways to address many of these issues in mixed design. REAP also participates in, designs, and advocates evaluations that use mixed design. 2. It should be noted that having relationships with local policy-makers can be tricky. Policy-makers are expecting their programmes/policies to work. If the evaluation shows success, it is easy. If the evaluation shows the programme/project is not successful, there may be friction. The key is to make sure that policy-makers understand the ground rules in advance. It is also helpful to develop a careful logic chain analysis plan in order to try to help policy-makers understand why a certain programme did not work as originally designed. This is true in all projects. But, it is especially true in the case of collaborations between evaluation teams and policy-makers. 3. We know cases when our advice has been heard and ‘accepted’ by the State Council but rejected by the Ministry of Education. When this happened, we saw no policy change. For example, we wrote a brief on financial aid for college for poor students in poor rural areas. In China all financial aid for college is given after the student enters college. Therefore, when the student is making his/her college decision, poor students do not know whether they will receive aid or not. Because of this we showed that poor students tended to pick majors that had tuition waivers (in exchange for postgraduation obligations – for example, teaching majors do not pay tuition and fees but they must teach in rural schools for five to seven years). We showed that if poor students were given financial aid offers before the college decision was made, they would change their decision-making process a lot. In other words, the current financial aid system distorts the decisions of those that need the aid the most. We suggested giving aid decisions out earlier. The Ministry of Education (for many possible reasons) objected. Until now there has not been any change.