Testing Differences Statistically with the Leiden Ranking

Loet Leydesdorff,Lutz Bornmann

DOI: https://doi.org/10.1007/s11192-012-0636-6

IF: 3.801

2011-01-01

Scientometrics

Abstract:The Leiden ranking 2011/2012 provides the Proportion top-10% publications (PP(top-10%)) as a new indicator. This indicator allows for testing performance differences between two universities for statistical significance.

What problem does this paper attempt to address?

Statistical Significance and Effect Sizes of Differences among Research Universities at the Level of Nations and Worldwide based on the Leiden Rankings

Loet Leydesdorff,Lutz Bornmann,John Mingers

DOI: https://doi.org/10.48550/arXiv.1710.11020

2018-10-13

Abstract:The Leiden Rankings can be used for grouping research universities by considering universities which are not statistically significantly different as homogeneous sets. The groups and intergroup relations can be analyzed and visualized using tools from network analysis. Using the so-called "excellence indicator" PPtop-10%--the proportion of the top-10% most-highly-cited papers assigned to a university--we pursue a classification using (i) overlapping stability intervals, (ii) statistical-significance tests, and (iii) effect sizes of differences among 902 universities in 54 countries; we focus on the UK, Germany, Brazil, and the USA as national examples. Although the groupings remain largely the same using different statistical significance levels or overlapping stability intervals, these classifications are uncorrelated with those based on effect sizes. Effect sizes for the differences between universities are small (w <.2). The more detailed analysis of universities at the country level suggests that distinctions beyond three or perhaps four groups of universities (high, middle, low) may not be meaningful. Given similar institutional incentives, isomorphism within each eco-system of universities should not be underestimated. Our results suggest that networks based on overlapping stability intervals can provide a first impression of the relevant groupings among universities. However, the clusters are not well-defined divisions between groups of universities.

Digital Libraries
The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation

Ludo Waltman,Clara Calero-Medina,Joost Kosten,Ed C.M. Noyons,Robert J.W. Tijssen,Nees Jan van Eck,Thed N. van Leeuwen,Anthony F.J. van Raan,Martijn S. Visser,Paul Wouters

DOI: https://doi.org/10.48550/arXiv.1202.3941

2012-02-17

Abstract:The Leiden Ranking 2011/2012 is a ranking of universities based on bibliometric indicators of publication output, citation impact, and scientific collaboration. The ranking includes 500 major universities from 41 different countries. This paper provides an extensive discussion of the Leiden Ranking 2011/2012. The ranking is compared with other global university rankings, in particular the Academic Ranking of World Universities (commonly known as the Shanghai Ranking) and the Times Higher Education World University Rankings. Also, a detailed description is offered of the data collection methodology of the Leiden Ranking 2011/2012 and of the indicators used in the ranking. Various innovations in the Leiden Ranking 2011/2012 are presented. These innovations include (1) an indicator based on counting a university's highly cited publications, (2) indicators based on fractional rather than full counting of collaborative publications, (3) the possibility of excluding non-English language publications, and (4) the use of stability intervals. Finally, some comments are made on the interpretation of the ranking, and a number of limitations of the ranking are pointed out.

Digital Libraries
Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA

Loet Leydesdorff,Caroline S. Wagner,Lin Zhang

DOI: https://doi.org/10.2478/jdis-2021-0014

2021-03-02

Journal of Data and Information Science

Abstract:Abstract Purpose Building on Leydesdorff, Bornmann, and Mingers (2019), we elaborate the differences between Tsinghua and Zhejiang University as an empirical example. We address the question of whether differences are statistically significant in the rankings of Chinese universities. We propose methods for measuring statistical significance among different universities within or among countries. Design/methodology/approach Based on z -testing and overlapping confidence intervals, and using data about 205 Chinese universities included in the Leiden Rankings 2020, we argue that three main groups of Chinese research universities can be distinguished (low, middle, and high). Findings When the sample of 205 Chinese universities is merged with the 197 US universities included in Leiden Rankings 2020, the results similarly indicate three main groups: low, middle, and high. Using this data (Leiden Rankings and Web of Science), the z -scores of the Chinese universities are significantly below those of the US universities albeit with some overlap. Research limitations We show empirically that differences in ranking may be due to changes in the data, the models, or the modeling effects on the data. The scientometric groupings are not always stable when we use different methods. Practical implications Differences among universities can be tested for their statistical significance. The statistics relativize the values of decimals in the rankings. One can operate with a scheme of low/middle/high in policy debates and leave the more fine-grained rankings of individual universities to operational management and local settings. Originality/value In the discussion about the rankings of universities, the question of whether differences are statistically significant, has, in our opinion, insufficiently been addressed in research evaluations.

information science & library science
What proportion of excellent papers makes an institution one of the best worldwide? Specifying thresholds for the interpretation of the results of the SCImago Institutions Ranking and the Leiden Ranking

Lutz Bornmann,Felix Moya Anegón,Felix de Moya Anegón

DOI: https://doi.org/10.1002/asi.23047

2013-10-30

Journal of the Association for Information Science and Technology

Abstract:University rankings generally present users with the problem of placing the results given for an institution in context. Only a comparison with the performance of all other institutions makes it possible to say exactly where an institution stands. In order to interpret the results of the SCImago Institutions Ranking (based on Scopus data) and the Leiden Ranking (based on Web of Science data), in this study we offer thresholds with which it is possible to assess whether an institution belongs to the top 1%, top 5%, top 10%, top 25%, or top 50% of institutions in the world. The thresholds are based on the excellence rate or PPtop 10%. Both indicators measure the proportion of an institution's publications which belong to the 10% most frequently cited publications and are the most important indicators for measuring institutional impact. For example, while an institution must achieve a value of 24.63% in the Leiden Ranking 2013 to be considered one of the top 1% of institutions worldwide, the SCImago Institutions Ranking requires 30.2%.

information science & library science,computer science, information systems
Statistical tests and research assessments: A comment on Schneider (2012).

Lutz Bornmann,Loet Leydesdorff

DOI: https://doi.org/10.1002/asi.22860

2013-01-01

Journal of the American Society for Information Science and Technology

Abstract: In a recent presentation at the 17th International Conference on Science and Technology Indicators, Schneider (2012) criticised the proposal of Bornmann, de Moya Anegon, and Leydesdorff (2012) and Leydesdorff and Bornmann (2012) to use statistical tests in order to evaluate research assessments and university rankings. We agree with Schneider's proposal to add statistical power analysis and effect size measures to research evaluations, but disagree that these procedures would replace significance testing. Accordingly, effect size measures were added to the Excel sheets that we bring online for testing performance differences between institutions in the Leiden Ranking and the SCImago Institutions Ranking.
Universitaire En Economische Pieken (mountains and Valleys in University and Economics Research in the Netherlands)

Wilfred Dolfsma,Loet Leydesdorff

2016-01-01

Abstract:Dutch Abstract: Nederlands onderzoek scoort hoog in de top-10% van meest geciteerde publicaties (Leydesdorff, Wagner en Bornmann, 2014). De Leidse Rankings van het Centrum voor Wetenschap en Technologiestudies (CWTS) stellen ons in staat dit per universiteit te beoordelen. Er bestaan statistisch significante verschillen tussen de Nederlandse universiteiten, maar niet tussen de economische instituten. De ranking op basis van de top-10% is negatief gecorreleerd aan die van de Institutentop van ESB. English Abstract: Scientific publications with a Dutch address are represented above expectation in the top-10% most cited publications worldwide (Leydesdorff, Wagner, & Bornmann, 2014). The Leiden Rankings enable us to test statistically (a) whether a score is significantly above expectations and (b) whether differences among nations, universities, faculties, etc., are statistically significant. Using universities, a division between the universities in the western part of the Netherlands (“Randstad”) and the more peripheral universities (e.g., Groningen) is indicated. When focusing on the economic faculties and institutes, however, this division is not found: smaller units seem to outperform the large faculties. Our ranking based on the top-10% most cited publications correlates negatively with the yearly one of ESB based on “article influence scores.” Rankings thus are highly sensitive for parameter choices.
The New Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011

Lutz Bornmann,Felix de Moya Anegon,Loet Leydesdorff

DOI: https://doi.org/10.1016/j.joi.2011.11.006

IF: 3.7

2012-01-01

Journal of Informetrics

Abstract:The new excellence indicator in the World Report of the SCImago Institutions Rankings (SIR) makes it possible to test differences in the ranking in terms of statistical significance. For example, at the 17th position of these rankings, UCLA has an output of 37,994 papers with an excellence indicator of 28.9. Stanford University follows at the 19th position with 37,885 papers and 29.1 excellence, and z = - 0.607. The difference between these two institution thus is not statistically significant. We provide a calculator at http://www.leydesdorff.net/scimago11/scimago11.xls in which one can fill out this test for any two institutions and also for each institution on whether its score is significantly above or below expectation (assuming that 10 papers are for stochastic reasons in the top-10
Percentile ranks and the integrated impact indicator (I3)

Loet Leydesdorff,Lutz Bornmann

DOI: https://doi.org/10.1002/asi.22641

2012-01-01

Journal of the American Society for Information Science and Technology

Abstract: We tested Rousseau's (in press) recent proposal to define percentile classes in the case of the Integrated Impact Indicator (I3) so that the largest number in a set always belongs to the highest (100th) percentile rank class. In the case a set of nine uncited papers and one with citation, however, the uncited papers would all be placed in the 90th percentile rank. A lowly-cited document set would thus be advantaged when compared with a highly-cited one. Notwithstanding our reservations, we extended the program for computing I3 in Web-of-Science data (at http://www.leydesdorff.net/software/i3) with this option; the quantiles without a correction are now the default. As Rousseau mentions, excellence indicators (e.g., the top-10%) can be considered as special cases of I3: only two percentile rank classes are distinguished for the evaluation. Both excellence and impact indicators can be tested statistically using the z-test for independent proportions.
A Longitudinal Analysis of University Rankings

Friso Selten,Cameron Neylon,Chun-Kai Huang,Paul Groth

DOI: https://doi.org/10.1162/qss_a_00052

2020-01-20

Abstract:Pressured by globalization and the increasing demand for public organisations to be accountable, efficient and transparent, university rankings have become an important tool for assessing the quality of higher education institutions. It is therefore important to carefully assess exactly what these rankings measure. In this paper, the three major global university rankings, The Academic Ranking of World Universities, The Times Higher Education and the Quacquarelli Symonds World University Rankings, are studied. After a description of the ranking methodologies, it is shown that university rankings are stable over time but that there is variation between the three rankings. Furthermore, using Principal Component Analysis and Exploratory Factor Analysis, we show that the variables used to construct the rankings primarily measure two underlying factors: a universities reputation and its research performance. By correlating these factors and plotting regional aggregates of universities on the two factors, differences between the rankings are made visible. Last, we elaborate how the results from these analysis can be viewed in light of often voiced critiques of the ranking process. This indicates that the variables used by the rankings might not capture the concepts they claim to measure. Doing so the study provides evidence of the ambiguous nature of university ranking's quantification of university performance.

Social and Information Networks,Digital Libraries,Physics and Society
Universitaire En Economische Pieken

Wilfred Dolfsma,Loet Leydesdorff

2016-01-01

Abstract:Nederlands onderzoek scoort hoog in de top-10% van meest geciteerde publicaties (Leydesdorff, Wagner en Bornmann, 2014). De Leidse Rankings van het Centrum voor Wetenschap en Technologiestudies (CWTS) stellen ons in staat dit per universiteit te beoordelen. Er bestaan statistisch significante verschillen tussen de Nederlandse universiteiten, maar niet tussen de economische instituten. De ranking op basis van de top-10% is negatief gecorreleerd aan die van de Institutentop van ESB.
A Heuristic Approach Based on Leiden Rankings to Identify Outliers: Evidence from Italian Universities in the European Landscape

Cinzia Daraio,Simone Di Leo,Loet Leydesdorff

DOI: https://doi.org/10.1007/s11192-022-04551-y

IF: 3.801

2022-01-01

Scientometrics

Abstract:We propose an innovative use of the Leiden Rankings (LR) in institutional management. Although LR only consider research output of major universities reported in Web of Science (WOS) and share the limitations of other existing rankings, we show that they can be used as a base of a heuristic approach to identify "outlying" institutions that perform significantly below or above expectations. Our approach is a non-rigorous intuitive method ("heuristic") because is affected by all the biases due to the technical choices and incompleteness that affect the LR but offers the possibility to discover interesting findings to be systematically verified later. We propose to use LR as a departure base on which to apply statistical analysis and network mapping to identify "outlier" institutions to be analyzed in detail as case studies. Outliers can inform and guide science policies about alternative options. Analyzing the publications of the Politecnico di Bari in more detail, we observe that "small teams" led by young and promising scholars can push the performance of a university up to the top of the LR. As argued by Moed (Applied evaluative informetrics. Springer International Publishing, Berlin, 2017a), supporting "emerging teams", can provide an alternative to research support policies, adopted to encourage virtuous behaviours and best practices in research. The results obtained by this heuristic approach need further verification and systematic analysis but may stimulate further studies and insights on the topics of university rankings policy, institutional management, dynamics of teams, good research practice and alternative funding methods.
A comparison of university performance scores and ranks by MNCS and FSS

Giovanni Abramo,Ciriaco Andrea D'Angelo

DOI: https://doi.org/10.1016/j.joi.2016.07.004

2018-10-30

Abstract:In a previous article of ours, we explained the reasons why the MNCS and all similar per-publication citation indicators should not be used to measure research performance, whereas efficiency indicators (output to input) such as the FSS are valid indicators of performance. The problem frequently indicated in measuring efficiency indicators lies in the availability of input data. If we accept that such data are inaccessible, and instead resort to per-publication citation indicators, the question arises as to what extent institution performance rankings by MNCS are different from those by FSS (and so what effects such results could have on policy-makers, managers and other users of the rankings). Contrasting the 2008-2012 performance by MNCS and FSS of Italian universities in the Sciences, we try to answer that question at field, discipline, and overall university level. We present the descriptive statistics of the shifts in rank, and the correlations of both scores and ranks. The analysis reveals strong correlations in many fields but weak correlations in others. The extent of rank shifts is never negligible: a number of universities shift from top to non-top quartile ranks.

Digital Libraries
The substantive and practical significance of citation impact differences between institutions: Guidelines for the analysis of percentiles using effect sizes and confidence intervals

Richard Williams,Lutz Bornmann

DOI: https://doi.org/10.48550/arXiv.1404.3720

2014-04-12

Abstract:In our chapter we address the statistical analysis of percentiles: How should the citation impact of institutions be compared? In educational and psychological testing, percentiles are already used widely as a standard to evaluate an individual's test scores - intelligence tests for example - by comparing them with the percentiles of a calibrated sample. Percentiles, or percentile rank classes, are also a very suitable method for bibliometrics to normalize citations of publications in terms of the subject category and the publication year and, unlike the mean-based indicators (the relative citation rates), percentiles are scarcely affected by skewed distributions of citations. The percentile of a certain publication provides information about the citation impact this publication has achieved in comparison to other similar publications in the same subject category and publication year. Analyses of percentiles, however, have not always been presented in the most effective and meaningful way. New APA guidelines (American Psychological Association, 2010) suggest a lesser emphasis on significance tests and a greater emphasis on the substantive and practical significance of findings. Drawing on work by Cumming (2012) we show how examinations of effect sizes (e.g. Cohen's d statistic) and confidence intervals can lead to a clear understanding of citation impact differences.

Digital Libraries
Expected values in percentile indicators

Lutz Bornmann,Robin Haunschild

DOI: https://doi.org/10.48550/arXiv.1611.05206

2016-11-16

Abstract:PP(top x%) is the proportion of papers of a unit (e.g. an institution or a group of researchers), which belongs to the x% most frequently cited papers in the corresponding fields and publication years. It has been proposed that x% of papers can be expected which belongs to the x% most frequently cited papers. In this Letter to the Editor we will present the results of an empirical test whether we can really have this expectation and how strong the deviations from the expected values are when many random samples are drawn from the database.

Digital Libraries
Revisiting Relative Indicators and Provisional Truths.

Loet Leydesdorff,Tobias Opthof

DOI: https://doi.org/10.48550/arxiv.1808.09665

2018-01-01

Abstract:Following discussions in 2010 and 2011, scientometric evaluators have increasingly abandoned relative indicators in favor of comparing observed with expected citation ratios. The latter method provides parameters with error values allowing for the statistical testing of differences in citation scores. A further step would be to proceed to non-parametric statistics (e.g., the top-10 distributions. In response to a plea for returning to relative indicators in the previous issue of this newsletter, we argue in favor of further progress in the development of citation impact indicators.
In Praise of Bibliometrics

Loet Leydesdorff

DOI: https://doi.org/10.1016/j.joi.2017.03.002

IF: 3.7

2017-01-01

Journal of Informetrics

Abstract:In this study, we examine academic ranking and inequality in library and information science (LIS) using a faculty hiring network of 643 faculty members from 44 LIS schools in the United States. We employ four groups of measures to study academic ranking, including adjacency, placement and hiring, distance-based measures, and hubs and authorities. Among these measures, closeness and hub measures have the highest correlation with the U.S. News ranking (r = 0.78). We study academic inequality using four distinct methods that include downward/upward placement, Lorenz curve, cliques, and egocentric networks of LIS schools and find that academic inequality exists in the LIS community. We show that the percentage of downward placement (68%) is much higher than that of upward placement (22%); meanwhile, 20% of the 30 LIS schools that have doctoral programs produced nearly 60% of all LIS faculty, with a Gini coefficient of 0.53. We also find cliques of highly ranked schools and a core/periphery structure that distinguishes LIS schools of different ranks. Overall, LIS faculty hiring networks have considerable value in deriving credible academic ranking and revealing faculty exchange within the field.
Turning the tables in citation analysis one more time: Principles for comparing sets of documents

Loet Leydesdorff,Lutz Bornmann,Rüdiger Mutz,Tobias Opthof

DOI: https://doi.org/10.48550/arXiv.1101.3863

2011-06-19

Abstract:We submit newly developed citation impact indicators based not on arithmetic averages of citations but on percentile ranks. Citation distributions are-as a rule-highly skewed and should not be arithmetically averaged. With percentile ranks, the citation of each paper is rated in terms of its percentile in the citation distribution. The percentile ranks approach allows for the formulation of a more abstract indicator scheme that can be used to organize and/or schematize different impact indicators according to three degrees of freedom: the selection of the reference sets, the evaluation criteria, and the choice of whether or not to define the publication sets as independent. Bibliometric data of seven principal investigators (PIs) of the Academic Medical Center of the University of Amsterdam is used as an exemplary data set. We demonstrate that the proposed indicators [R(6), R(100), R(6,k), R(100,k)] are an improvement of averages-based indicators because one can account for the shape of the distributions of citations over papers.

Digital Libraries,Physics and Society
Severe Language Effect in University Rankings: Particularly Germany and France are wronged in citation-based rankings

Anthony F.J. van Raan,Thed N. van Leeuwen,Martijn S. Visser

DOI: https://doi.org/10.48550/arXiv.1012.5199

2010-12-23

Abstract:We applied a set of standard bibliometric indicators to monitor the scientific state-of-arte of 500 universities worldwide and constructed a ranking on the basis of these indicators (Leiden Ranking 2010). We find a dramatic and hitherto largely underestimated language effect in the bibliometric, citation-based measurement of research performance when comparing the ranking based on all Web of Science (WoS) covered publications and on only English WoS covered publications, particularly for Germany and France.

Digital Libraries,Physics and Society
Turning the tables on citation analysis one more time: Principles for comparing sets of documents

Loet Leydesdorff,Lutz Bornmann,Rüdiger Mutz,Tobias Opthof

DOI: https://doi.org/10.1002/asi.21534

2011-01-01

Journal of the American Society for Information Science and Technology

Abstract:We submit newly developed citation impact indicators based not on arithmetic averages of citations but on percentile ranks. Citation distributions are—as a rule—highly skewed and should not be arithmetically averaged. With percentile ranks, the citation score of each paper is rated in terms of its percentile in the citation distribution. The percentile ranks approach allows for the formulation of a more abstract indicator scheme that can be used to organize and/or schematize different impact indicators according to three degrees of freedom: the selection of the reference sets, the evaluation criteria, and the choice of whether or not to define the publication sets as independent. Bibliometric data of seven principal investigators (PIs) of the Academic Medical Center of the University of Amsterdam are used as an exemplary dataset. We demonstrate that the proposed family indicators [R(6), R(100), R(6, k), R(100, k)] are an improvement on averages-based indicators because one can account for the shape of the distributions of citations over papers. © 2011 Wiley Periodicals, Inc.
The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits

Lutz Bornmann,Loet Leydesdorff,Ruediger Mutz

DOI: https://doi.org/10.48550/arXiv.1211.0381

2012-11-02

Abstract:Percentiles have been established in bibliometrics as an important alternative to mean-based indicators for obtaining a normalized citation impact of publications. Percentiles have a number of advantages over standard bibliometric indicators used frequently: for example, their calculation is not based on the arithmetic mean which should not be used for skewed bibliometric data. This study describes the opportunities and limits and the advantages and disadvantages of using percentiles in bibliometrics. We also address problems in the calculation of percentiles and percentile rank classes for which there is not (yet) a satisfactory solution. It will be hard to compare the results of different percentile-based studies with each other unless it is clear that the studies were done with the same choices for percentile calculation and rank assignment.

Digital Libraries,Applications

Testing Differences Statistically with the Leiden Ranking

Statistical Significance and Effect Sizes of Differences among Research Universities at the Level of Nations and Worldwide based on the Leiden Rankings

The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation

Are University Rankings Statistically Significant? A Comparison among Chinese Universities and with the USA

What proportion of excellent papers makes an institution one of the best worldwide? Specifying thresholds for the interpretation of the results of the SCImago Institutions Ranking and the Leiden Ranking

Statistical tests and research assessments: A comment on Schneider (2012).

Universitaire En Economische Pieken (mountains and Valleys in University and Economics Research in the Netherlands)

The New Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011

Percentile ranks and the integrated impact indicator (I3)

A Longitudinal Analysis of University Rankings

Universitaire En Economische Pieken

A Heuristic Approach Based on Leiden Rankings to Identify Outliers: Evidence from Italian Universities in the European Landscape

A comparison of university performance scores and ranks by MNCS and FSS

The substantive and practical significance of citation impact differences between institutions: Guidelines for the analysis of percentiles using effect sizes and confidence intervals

Expected values in percentile indicators

Revisiting Relative Indicators and Provisional Truths.

In Praise of Bibliometrics

Turning the tables in citation analysis one more time: Principles for comparing sets of documents

Severe Language Effect in University Rankings: Particularly Germany and France are wronged in citation-based rankings

Turning the tables on citation analysis one more time: Principles for comparing sets of documents

The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits