Selecting Criterion and Limitation of Policy Evaluation Model

Jiasu Lei

2008-01-01

Abstract:Based on a review and definition of the purpose,definition and extension of policy evaluation,this paper discussed the selecting criterions of policy evaluation model.Shortages of existing models are summarized and some advices are given,too.Hope this research can be useful for future evaluations and promote the creation of new models.

What problem does this paper attempt to address?

A Dilemma of Multi-model of Public Policy Evaluation and A Solution Based on Philosophical Thinking

Zhou Jianguo

2012-01-01

Abstract:Through historical inspection of the development of public policy evaluation model,this paper analyzes the logic of the transformation from policy single model to multi-model and the dilemma of multi-model of public policy evaluation.Based on the case of migrant policy evaluation in South-to-North Water Diversion,the paper explained in pluralistic rationality view the inaccurate limit to plural benefits,and the current configuration solidification of evaluation model and so on.For basic solution of pluralistic evaluation,we must further transform the philosophical thought of policy evaluation and this transformation must follow the core values of the critical policy analysis theory,the stakeholder theory as well as the rational choice theory.
What evaluation criteria are used in policy evaluation research: A cross-field literature review

Céline Mavrot,Oto Potluka,Lars Balzer,Véronique Eicher,Sigrid Haunberger,Christine Heuer,François-Xavier Viallon

DOI: https://doi.org/10.1016/j.evalprogplan.2024.102512

2024-10-10

Abstract:This literature review offers a comprehensive overview of the use of evaluation criteria across five policy fields: social services, land-use planning, teaching in higher education, vocational education, and the environment. Though it is a key part of the evaluation process, the question of how criteria are defined, chosen, and applied generates surprisingly little debate among the evaluation community. In evaluation practice, criteria are often taken for granted - and occasionally even used in ways that are neither explicit nor transparent. This cross-field literature review shows a strong presence of routinized evaluation criteria (relating to the specifics of each policy field), while some new sets of higher-degree criteria also emerge in the face of social challenges relating to sustainability, public acceptance, or social justice. Criteria development draws on both inductive bottom-up processes (which can include policy stakeholders) and top-down deductive processes (which derive criteria from the literature, as well as from national and international standards). A more profound reflection on evaluation criteria (that is, the dimensions used by societies to assess the success of policy interventions) might be required in the future of evaluation research and planning; a deeper cross-field dialogue could support this endeavor.
Review on the Concept,Model,Development History and Methods of Library Evaluation

Zhang Guangqin

DOI: https://doi.org/10.3969/j.issn.1002-1027.2011.03.001

2011-01-01

Abstract:From the perspective of the quantity and subject distributions of the research on library evaluation,the framework and comparison of the concepts and the three conceptive models of library evaluation are discussed.The first one is the model regarding the overall library as an assessment unit;the second one is the model focusing on certain type or stage of the library activities,and the last one is the PDCA model.This paper also discusses the historical development and the status of the library evaluation study.
An Analysis on Logic and Model of Public Policy Design

WANG Hong-bo,LIU Min

DOI: https://doi.org/10.3969/j.issn.1008-245x.2006.01.001

2006-01-01

Abstract:From the analysis on the logic structure of public policies and on the basis of building the synthetic tub curves,the paper brings up an analytic model of the policy function,a analytic model for policy time and space,and formulating of assorted policies and substitution policies to make up the policy effectiveness at the stage of the basic policy invalidation.It furthermore presents a brand-new decision-making tree approach for public policies,and introduces factors like public response probability,expert evaluation and governmental weighting coefficients,etc.to impel the scientific reform form formulating policies and systems and choosing decision-making modes.
Cost-effective Use of Evaluation Models: an Empirical Cross-National Investigation

WM de Jong

DOI: https://doi.org/10.1504/ijtm.2000.002821

2000-01-01

International Journal of Technology Management

Abstract:Despite the existence of methodologically sound evaluation models for infrastructure projects, it is far from certain if these methods are used at all. In fact, political motives seem often to play a more important role when it comes to "real decision-making". This paper provides an answer to the question of why certain evaluation methods are used as the basis to allocate funds and others are not, even if they are qualitatively of high standing. The evaluation methods used in six different countries and corresponding authorities (France, Germany, Switzerland, Netherlands, UK and the MTC/Bay Area) have been analysed. As a result, seven explanatory factors which determine the effective use of appraisal methods have been derived: range of different criteria, understanding and dispersion, contributions of lower government tiers, contribution of pressure groups, agreement among actors on usage, flexibility of usage, and legal recognition. A country-specific discussion is carried out in terms of these seven factors. The paper closes with a cross-national comparison and practical considerations for successful implementation of evaluation methods.
Development of the theoretical model on knowledge translation from health technology assessment to health policy in China

CHEN Ying-yao,MAO Yi-wei,TANG Meng,SHI Li-zheng,DONG Heng-jin

DOI: https://doi.org/10.3969/j.issn.1674-2982.2013.07.002

2013-01-01

Abstract:Many studies of health technology assessment ( HTA) have been carried out to evaluate policy changes associated with ChinaK's health reform. How to successfully use the knowledge generated by such studies to inform health policy and decision-making is an important issue. By reviewing the definition of knowledge translation and three knowledge translation models that are commonly used,this study has proposed a theoretical model of knowledge translation from health technology assessment to health policy in China,which focuses on the influencing factors in the aspects of researchers and policymakers. Furthermore,this study has provided some recommendations on promoting HTA-related knowledge translation,such as enhancing communication between the researchers and policymakers,establishing incentive mechanisms to encourage knowledge translation.
Constructing an Integrated Stakeholder-Based Participatory Policy Evaluation Model for Urban Traffic Restriction

Shi Jia,Lee Ching-Hung,Xuesong Guo,Zhu Zhengwei

DOI: https://doi.org/10.1016/j.techfore.2019.119839

IF: 12

2020-01-01

Technological Forecasting and Social Change

Abstract:Effectively evaluating public policy based on diverse public interests and requirements under the incremental social change in a developing city is a critical challenge. Fairness and an objective policy evaluation process need to be considered to deliver the expected policy value with large-scale population concerns. This study attempts to propose a novel stakeholder-based policy evaluation model to drive the enhancement of policy evaluation through three phases, that is, (1) policy document analysis, (2) evaluators identification, and (3) participatory policy evaluation. In Phase 1, a selection of policy measures and policy stakeholders' identification are analysed using a document analysis approach. In Phase 2, evaluators identification procedures are adopted to identify key participants through social network analysis (SNA). Finally, AHP-fuzzy comprehensive evaluation is used in the third phase to better evaluate the policy. Then, a comprehensive case study of the urban traffic restriction policy of Xi'an City is demonstrated to verify this approach. Public participation and the indirect effect are revealed as vital factors that impact the effectiveness and vitality of public policy development. The implications are discussed to obtain valuable insights for suggestions on future policies.
Understanding Evaluation Policy and Organizational Capacity for Evaluation: An Interview Study

Hind Al Hudib,J. Bradley Cousins

DOI: https://doi.org/10.1177/10982140211007573

2021-07-15

American Journal of Evaluation

Abstract:Evaluation policy has been identified as an important means of shaping and influencing organizational evaluation practice, yet, to date, little empirical research has been conducted to deepen our understanding of this relationship. The purpose of this study was to illuminate evaluation policy’s role in leveraging organizational capacity to do and use evaluation. We interviewed 18 published evaluation scholars and practitioners from North America and Europe about this topic. A thematic analysis of findings underscores the importance of context, policy attributes, enablers, and organizational benefits. Based on the findings, we developed an ecological conceptual framework to guide thinking about the role of evaluation policy in capacity building. We discuss these findings in terms of practical implications of understanding context, redressing the imbalance between learning and accountability purposes of evaluation, and organizational leadership, and we conclude with some implications for research.

social sciences, interdisciplinary
Research on The Theoretical Basis and Evolution of Discrete Choice Models

聂冲,贾生华

DOI: https://doi.org/10.3969/j.issn.1000-3894.2005.11.016

2005-01-01

Abstract:The boom of discrete choice models , which belong to the area of microeconometrics,were really beginning from the end of 1950s.The models can be used to analyse the behaviors of private and family experientially, and they are widely applied to economics and other social sciences. This paper begins with the theoretical basis and utility maximization, then demonstrates that there are three limitations of logit: it can not represent random taste variation; it implies proportional substitution across alternatives and cannot handle situations where unobserved factors are correlated over time, though logit is by far the easiest and most widely used discrete choice model. Other models, such as GEV (including nested logit), probit and mixed logit, have arisen largely to avoid these limitations.
A Study on the Evaluation Model for Outstanding Scholars in Humanities and Social Sciences

WANG Zhou-yi,GUO Lin,LI Jing

DOI: https://doi.org/10.16365/j.cnki.11-4054/d.2011.01.025

2011-01-01

Abstract:Taking the selection of Academician in humanities and social sciences in 1948 as a reference,this paper analyzes existing various evaluation model for outstanding scholars in humanities and social sciences and holds that Academician in humanities and social sciences should be selected in the way of peer review organized by authoritative academic institutions.
Analysis of the Theory Foundation and Patterns of Performance Evaluation on the Government's Science and Technology Program

谈毅,仝允桓

DOI: https://doi.org/10.3969/j.issn.1003-2053.2004.02.008

2004-01-01

Abstract:The performance evaluation on the science and technology program has been the keystone of the innovative policies in more and more countries.It is partly due to the tight budget leading to the emphasis on the rational allocation of the limited public research fund.And in the policy trend giving the priority to the innovation, it is more important to evaluate the outcomes of the relevant programs or projects.With the performance evaluation,the science and technology policy-making process is more accountability,transparency and less distorted intervention.On the basis of the current literature, the paper analyzes the theory foundation and the practice of the performance evaluation on the science and technology program in order to improve the system of China's science and technology policies.
Construction on Performance Evaluation System of Financial Expenditure Based on Logic Model: Taking the Agricultural Expenditure as an Example

Wu Jian

Abstract:Evaluation index is the core of expenditure performance evaluation.But now,the evaluation index less focus on evaluating the whole process of expenditure,and it is also lack of a systematic method and cannot reflect the real status of the government's financial expenditure performance completely.Leading the logic model into the index design,we design a systematic evaluation index with four aspects: 'Input—Output—Medium term outcome—Long term outcome'.This method can incarnate the economic,efficient and effective principle of the performance evaluation better.

Agricultural and Food Sciences,Economics
The Empirical Study For Evaluation Lndexes Of It Project Success Criterion

Xiangnan Lu,Yuxiao Jiao,Huirong Li

2007-01-01

Abstract:In the modem society, market competitions for IT projects become more and more fiercely. Therefore, information technique becomes more and more important to enterprises. Thus, the, research on excellent IT projects success criterion is very useful and many scholars do researches about it. This paper tries to give a model for the evaluation indexes of IT projects success criterion. It does some researches and discusses the different views of the evaluation indexes of projects success criterion and then gives a success criterion evaluation model for IT projects. After the questionnaire research, reliability and validity analysis, descriptive analysis, and variance analysis of the sample data, the model is revised to a better one. And in the end of the paper the author draws some conclusions.
The opportunities and challenges of health program evaluation

Lei Peng,Wu Zhuo-chun

DOI: https://doi.org/10.3969/j.issn.1007-953X.2009.02.020

2009-01-01

Abstract:The main purpose of health program evaluation is to provide information for decision making and efficiently allocate health resources.The rapid development of health care sector in recent decades brings big opportunity for health program evaluation.Meanwhile,the theory and methodology of evaluation lag behind the practice of health program evaluation,so we think it is necessary to make thorough exploration on oppotrunities and challenges of health program evaluation on its theory,methodology and practice.
Study on Performance Evaluation Model and Index Systems of Administrative Service Centers

Shuo Yang,Han Li,Li-Li Cao

DOI: https://doi.org/10.2991/icmesd-17.2017.74

2017-01-01

Abstract:In this paper, we analyzed the status quo and existing problems of performance evaluation. Using KPI index design method, we constructed a performance evaluation model of administrative service according to the layers of administrative service activities. Such work has laid a foundation for putting theoretical research on performance evaluation of administrative service center into practical application, which can facilitate the quality promotion and development of administrative service centers.
Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?

Xueru Wen,Jie Lou,Yaojie Lu,Hongyu Lin,Xing Yu,Xinyu Lu,Ben He,Xianpei Han,Debing Zhang,Le Sun

2024-10-15

Abstract:Reward Models (RMs) are crucial for aligning language models with human preferences. Currently, the evaluation of RMs depends on measuring accuracy against a validation set of manually annotated preference data. Although this method is straightforward and widely adopted, the relationship between RM accuracy and downstream policy performance remains under-explored. In this work, we conduct experiments in a synthetic setting to investigate how differences in RM measured by accuracy translate into gaps in optimized policy performance. Our findings reveal that while there is a weak positive correlation between accuracy and downstream performance, policies optimized towards RMs with similar accuracy can exhibit quite different performance. Moreover, we discover that the way of measuring accuracy significantly impacts its ability to predict the final policy performance. Through the lens of Regressional Goodhart's effect, we identify the existence of exogenous variables impacting the relationship between RM quality measured by accuracy and policy model capability. This underscores the inadequacy of relying solely on accuracy to reflect their impact on policy optimization.

Machine Learning,Artificial Intelligence,Computation and Language
Research on an Evaluation Model of Aerospace Reconnaissance Intelligence Support Capacity

Yaqun Yi

2012-01-01

Abstract:The paper builds a synthetic evaluation index system of aerospace reconnaissance intelligence support capacity,develops a synthetic evaluation model,quantizing and synthesizing the inherent information of the assessing index system of aerospace reconnaissance intelligence support capacity,and the subjective judgment of evaluator,based on improved entropy method.A study case and its analysis result prove that the model and the solution method are useful and effective.
Theory and Methods on the Construction of Core Public Policy Evaluation Index System

杨存,高羽,陈功,郑晓瑛

DOI: https://doi.org/10.3969/j.issn.1003-0743.2011.08.010

2011-01-01

Abstract:Objective: To give a summary of the theory and methods on the construction of core public policy evaluation index system to show a clear idea of the construction of the evaluation index system for public policy.Methods: Using the evaluation of government performance,project management evaluation and other related disciplines of performance evaluation,and combining with the experience of performance evaluation in medical area,the quantitative statistical methods are summarized.Results: The evaluation of public policy's complexity is with its diversity.It usually cannot find the effect of public policy system with very few significant indicators,but the ideas,theories and methods are similarity.Conclusion: The same thing happens in the ongoing health care reform in China.Facing the new challenges,the ideas and methods can be applied in the field of medical and health system for monitoring and evaluation with multi-point attempts.
Process models of governmental resource allocation and program evaluation

Patrick D. Larkey

DOI: https://doi.org/10.1007/bf01719636

IF: 5.3

1977-09-01

Policy Sciences

Abstract:The General Revenue Sharing program poses problems for systematic evaluation that are similar to the problems posed by many public programs. Evaluation is very difficult when programs are not planned experiments and when program effects depend upon the discretionary responses of many individuals or organizations. The main difficulty is in knowing what would have happened without the program. Models of behavioral processes have an indispensable role in evaluating such programs. This paper considers the role of “process models” in program evaluation. It summarizs research on the impact of General Revenue Sharing on municipal fiscal behavior that used models of municipal resource allocation processes. Quasi-experimental research designs have been characterized as inherently inferior to experimental designs for program evaluation. There is no inherent inferiority. When we have good descriptive models of behavior, it is possible to evaluate programs rigorously without classic experimental research designs.

public administration,social sciences, interdisciplinary
Study on Evaluation Model of State－owned Asset Performance and Its Application

Liang Liu

2000-01-01

Abstract:The evaluation model of state－owned asset performance is established by mathematicalmethods. In the model 14 indices are selected,which cover the benefit of asset management, the quality of asset, the ability of debt payment, the ability of development and social contribution.This model also incorporates factors as market shares, human resources, welfare etc. This model has been applied to evaluate the performance of a certain listed company in China.

Selecting Criterion and Limitation of Policy Evaluation Model

A Dilemma of Multi-model of Public Policy Evaluation and A Solution Based on Philosophical Thinking

What evaluation criteria are used in policy evaluation research: A cross-field literature review

Review on the Concept,Model,Development History and Methods of Library Evaluation

An Analysis on Logic and Model of Public Policy Design

Cost-effective Use of Evaluation Models: an Empirical Cross-National Investigation

Development of the theoretical model on knowledge translation from health technology assessment to health policy in China

Constructing an Integrated Stakeholder-Based Participatory Policy Evaluation Model for Urban Traffic Restriction

Understanding Evaluation Policy and Organizational Capacity for Evaluation: An Interview Study

Research on The Theoretical Basis and Evolution of Discrete Choice Models

A Study on the Evaluation Model for Outstanding Scholars in Humanities and Social Sciences

Analysis of the Theory Foundation and Patterns of Performance Evaluation on the Government's Science and Technology Program

Construction on Performance Evaluation System of Financial Expenditure Based on Logic Model: Taking the Agricultural Expenditure as an Example

The Empirical Study For Evaluation Lndexes Of It Project Success Criterion

The opportunities and challenges of health program evaluation

Study on Performance Evaluation Model and Index Systems of Administrative Service Centers

Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?

Research on an Evaluation Model of Aerospace Reconnaissance Intelligence Support Capacity

Theory and Methods on the Construction of Core Public Policy Evaluation Index System

Process models of governmental resource allocation and program evaluation

Study on Evaluation Model of State－owned Asset Performance and Its Application