Could Bibliometrics Reveal Top Science and Technology Achievements and Researchers? The Case for Evaluatology-based Science and Technology Evaluation

Guoxin Kang,Wanling Gao,Lei Wang,Chunjie Luo,Hainan Ye,Qian He,Shaopeng Dai,Jianfeng Zhan

2024-08-22

Abstract:By utilizing statistical methods to analyze bibliographic data, bibliometrics faces inherent limitations in identifying the most significant science and technology achievements and researchers. To overcome this challenge, we present an evaluatology-based science and technology evaluation methodology. At the heart of this approach lies the concept of an extended evaluation condition, encompassing eight crucial components derived from a field. We define four relationships that illustrate the connections among various achievements based on their mapped extended EC components, as well as their temporal and citation links. Within a relationship under an extended evaluation condition, evaluators can effectively compare these achievements by carefully addressing the influence of confounding variables. We establish a real-world evaluation system encompassing an entire collection of achievements, each of which is mapped to several components of an extended EC. Within a specific field like chip technology or open source, we construct a perfect evaluation model that can accurately trace the evolution and development of all achievements in terms of four relationships based on the real-world evaluation system. Building upon the foundation of the perfect evaluation model, we put forth four-round rules to eliminate non-significant achievements by utilizing four relationships. This process allows us to establish a pragmatic evaluation model that effectively captures the essential achievements, serving as a curated collection of the top N achievements within a specific field during a specific timeframe. We present a case study on the top 100 Chip achievements which highlights its practical application and efficacy in identifying significant achievements and researchers that otherwise can not be identified by using bibliometrics.

Computational Engineering, Finance, and Science,Computers and Society

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to more accurately identify the most significant scientific and technological achievements and researchers, and overcome the limitations of existing bibliometric methods**.

Specifically, the paper points out that although bibliometric methods have long relied on observable indicators (such as the number of publications, the number of citations, H - index, etc.) to assess the relevance and impact of academic achievements, these methods have three inherent limitations:

1. **The influence of various confounding variables**: For example, factors such as the reputation and network of researchers, and institutional background may lead to the artificial manipulation of the number of citations.
2. **Ignoring important non - bibliometric indicators**: For research that may not have a large number of publications but has significant technological achievements, bibliometric methods cannot effectively assess.
3. **Emphasizing quantity over quality**: Many bibliometric methods attach more importance to the quantity of publications rather than quality, which may lead to an incomplete assessment of the real value and influence of research achievements.

To overcome these limitations, the paper proposes a scientific and technological evaluation method based on Evaluatology. The core concept of this method is "Extended Evaluation Condition", which includes eight key components, covering all aspects from the research field to specific problem instances. By defining four relationships (pioneering, progressive, parallel, and related but unconnected relationships), evaluators can effectively compare different achievements while considering the influence of confounding variables.

The paper also proposes a four - round - rule method. By gradually eliminating non - important achievements, a practical evaluation model is established, and finally, the top achievements and researchers in a specific field are identified. The paper demonstrates the effectiveness and practicality of this method through a case study - the top 100 achievements in the chip technology field (Chip100).

In conclusion, this paper aims to provide a more comprehensive and accurate scientific and technological evaluation framework by introducing a new method based on Evaluatology to overcome the shortcomings of traditional bibliometric methods.