Introducing recalibrated academic performance indicators in the evaluation of individuals' research performance: A case study from Eastern Europe

Gyorgy Csomos
DOI: https://doi.org/10.48550/arXiv.2006.13321
2020-06-24
Abstract:In Hungary, the highest and most prestigious scientific qualification is considered to be the Doctor of Science (DSc) title being awarded by the Hungarian Academy of Sciences. The academic performance indicators of the DSc title are of high importance in the evaluation of individuals' research performance not only when a researcher applies for obtaining a DSc title, but also during promotions and appointments at universities, and in the case of the evaluation of applications for scientific titles and degrees, and the assessment of applications for funding. In the Section of Earth Sciences encompassing nine related disciplines, rather than carrying out a straightforward bibliometric analysis, the performance indicators were designed as a result of a consensual agreement between leading academicians, each of whom represented a particular discipline. Therefore, the minimum values of the indicators, required to be fulfilled if one is applying for a DSc title, do not adequately reflect the actual discipline-specific performance of researchers. This problem may generate tension between researchers during the evaluation process. The main goal of this paper is to recalibrate the minimum values of four major performance indicators by taking the actual discipline-specific distance ratios into account. In addition, each minimum value will be defined by employing integer and fractional counting methods as well. The research outcome of this study can provide impetus for the Section of Earth Sciences to optimize the minimum values of the DSc title performance indicators by taking the specifics of each discipline into account. Because academic performance indicators are also employed in other Eastern European countries in the evaluation of individuals' research performance, the methods used in that paper can be placed into a wider geographical context.
Digital Libraries
What problem does this paper attempt to address?
The problem that this paper attempts to solve is that when the Earth Sciences Department of the Hungarian Academy of Sciences (HAS) evaluates researchers' applications for the Doctor of Science (DSc) title, the academic performance indicators used do not fully reflect the specific characteristics and differences of various disciplines. Specifically: 1. **Limitations of the existing performance indicators**: The current minimum performance indicator values are reached by consensus among a small number of senior researchers, rather than based on rigorous bibliometric analysis. This has led to the actual publication characteristics and output differences between different disciplines not being properly considered, which may cause tensions among researchers. 2. **Neglect of discipline - specific distance ratios**: There are significant differences in the publication characteristics and scientific research outputs of different disciplines (such as geology, meteorology, social geography, etc.), but these differences are not fully reflected in the current performance indicators. For example, some disciplines place more emphasis on the number of journal articles, while others may attach more importance to the publication of books or monographs. 3. **Insufficient application of integer and fractional counting methods**: The existing performance indicators do not fully consider the different impacts of integer and fractional counting methods when calculating. This may lead to inaccurate assessment of the actual contributions of researchers. To address these problems, the main objective of the paper is to recalibrate the minimum values of four main performance indicators. The specific methods include: - **Introducing discipline - specific distance ratios**: Adjust the minimum values of performance indicators according to the actual publication characteristics and scientific research outputs of each discipline. - **Applying integer and fractional counting methods**: Ensure that when calculating performance indicators, the contributions of researchers can be more accurately reflected. Through this method, the paper hopes to provide a basis for optimizing the DSc - title performance indicators for the Earth Sciences Department of the Hungarian Academy of Sciences and other similar institutions, so as to evaluate the performance of researchers more fairly and accurately. In addition, since similar academic performance indicators are also used in other Eastern European countries, the research methods can be used for reference in a wider geographical area. ### Formula representation During the recalibration process, the involved formula can be expressed as: \[ \text{New performance indicator value} = f(\text{Discipline - specific distance ratio}, \text{Integer count}, \text{Fraction count}) \] where: - \( f \) is a function used to comprehensively consider the influence of discipline - specific distance ratios and different counting methods. - **Discipline - specific distance ratio**: Measures the relative differences in scientific research outputs between different disciplines. - **Integer count** and **Fraction count**: Represent different counting methods adopted when calculating performance indicators respectively. In this way, it is ensured that the new performance indicators can more accurately reflect the characteristics of each discipline and the actual contributions of researchers.