‘Hidden’ citations conceal the true impact of research

Michael Allen

DOI: https://doi.org/10.1088/2058-7058/37/07/13

2024-07-01

Physics World

Abstract:Papers introducing concepts that have since become common knowledge are often under-cited by researchers, skewing those articles’ true impact.

physics, multidisciplinary

What problem does this paper attempt to address?

Hidden Citations Obscure True Impact in Science

Xiangyi Meng,Onur Varol,Albert-László Barabási

2024-05-12

Abstract:References, the mechanism scientists rely on to signal previous knowledge, lately have turned into widely used and misused measures of scientific impact. Yet, when a discovery becomes common knowledge, citations suffer from obliteration by incorporation. This leads to the concept of hidden citation, representing a clear textual credit to a discovery without a reference to the publication embodying it. Here, we rely on unsupervised interpretable machine learning applied to the full text of each paper to systematically identify hidden citations. We find that for influential discoveries hidden citations outnumber citation counts, emerging regardless of publishing venue and discipline. We show that the prevalence of hidden citations is not driven by citation counts, but rather by the degree of the discourse on the topic within the text of the manuscripts, indicating that the more discussed is a discovery, the less visible it is to standard bibliometric analysis. Hidden citations indicate that bibliometric measures offer a limited perspective on quantifying the true impact of a discovery, raising the need to extract knowledge from the full text of the scientific corpus.

Computation and Language,Digital Libraries,Social and Information Networks,Physics and Society
Uncited articles and their effect on the concentration of citations

Diego Kozlowski1,Jens Peter Andersen,Vincent Larivière

DOI: https://doi.org/10.1002/asi.24852

2023-06-16

Abstract:Empirical evidence demonstrates that citations received by scholarly publications follow a pattern of preferential attachment, resulting in a power-law distribution. Such asymmetry has sparked significant debate regarding the use of citations for research evaluation. However, a consensus has yet to be established concerning the historical trends in citation concentration. Are citations becoming more concentrated in a small number of articles? Or have recent geopolitical and technical changes in science led to more decentralized distributions? This ongoing debate stems from a lack of technical clarity in measuring inequality. Given the variations in citation practices across disciplines and over time, it is crucial to account for multiple factors that can influence the findings. This article explores how reference-based and citation-based approaches, uncited articles, citation inflation, the expansion of bibliometric databases, disciplinary differences, and self-citations affect the evolution of citation concentration. Our results indicate a decreasing trend in citation concentration, primarily driven by a decline in uncited articles, which, in turn, can be attributed to the growing significance of Asia and Europe. On the whole, our findings clarify current debates on citation concentration and show that, contrary to a widely-held belief, citations are increasingly scattered.

Digital Libraries,Computers and Society,Physics and Society
Citing for High Impact

Xiaolin Shi,Jure Leskovec,Daniel A. McFarland

DOI: https://doi.org/10.48550/arXiv.1004.3351

2010-04-20

Abstract:The question of citation behavior has always intrigued scientists from various disciplines. While general citation patterns have been widely studied in the literature we develop the notion of citation projection graphs by investigating the citations among the publications that a given paper cites. We investigate how patterns of citations vary between various scientific disciplines and how such patterns reflect the scientific impact of the paper. We find that idiosyncratic citation patterns are characteristic for low impact papers; while narrow, discipline-focused citation patterns are common for medium impact papers. Our results show that crossing-community, or bridging citation patters are high risk and high reward since such patterns are characteristic for both low and high impact papers. Last, we observe that recently citation networks are trending toward more bridging and interdisciplinary forms.

Digital Libraries
Why There Are So Many Contradicted or Exaggerated Findings in Highly-Cited Clinical Research?

Mengyi Lu,Suyu Liu,Ying Yuan

DOI: https://doi.org/10.1016/j.cct.2022.106782

IF: 2.261

2022-01-01

Contemporary Clinical Trials

Abstract:It is not uncommon that clinical studies of the same intervention contradicted with each other, e.g., one study produced positive results, while the other produced negative results. Ioanndis (2005a) found that among 49 highly-cited original clinical research studies, published in New England Journal of Medicine, Journal of the American Medical Association, Lancet or in a high-impact medical specialty journal, 32% of them were contradicted in subsequent large-scale studies, or were shown to have potentially overestimated the efficacy of the experimental intervention. This finding is disturbing and of serious concern given the widespread impact of these highly-cited studies and the rigorous standards used to design and conduct the studies. We perform Bayesian analysis of these highly-cited clinical studies based on Bayesian factor. We identified one cause of the issue: p values strongly overstated the experimental evidence. For the highly-cited studies, when the p value was 0.05, there was a 74.4% percentage chance that the null hypothesis was true. The use of a p value of 0.05 as the criterion for significance caused many researchers to mistakenly draw conclusions of positive findings, which were then contradicted by subsequent large-scale studies.
Some Insights into the Factors Influencing Continuous Citation of Retracted Scientific Papers

Bor Luen Tang

DOI: https://doi.org/10.3390/publications11040047

2023-10-09

Publications

Abstract:Once retracted, the citation count of a research paper might be intuitively expected to drop precipitously. Here, we assessed the post-retraction citation of life and medical sciences papers from two top-ranked, multidisciplinary journals Nature and Science, from 2010 to 2018. Post-retraction citations accounted for a staggering 47.7% and 40.9% of total citations (median values), respectively, of the papers included in our analysis. These numbers are comparable with those from two journals with lower impact factors, and with retracted papers from the physical sciences discipline. A more qualitative assessment of five papers from the two journals with a high percentage (>50%) of post-retraction citations, all of which are associated with misconduct, reveal different contributing reasons and factors. Retracted papers associated with highly publicized misconduct cases are more prone to being cited with the retraction status indicated, or projected negatively (such as in the context of research ethics and misconduct discussions), with the latter also indicated by cross-disciplinary citations by humanities and social sciences articles. Retracted papers that retained significant validity in their main findings/conclusions may receive a large number of neutral citations that are somewhat blind to the retraction. Retracted papers in popular subject areas with massive publication outputs, particularly secondary publications such as reviews, may also have a high background citation noise. Our findings add further insights to the nature of post-retraction citations beyond the plain notion that these are largely made through sheer ignorance or negligence by the citing authors.
Exclusive: the papers that most heavily cite retracted studies

Richard Van Noorden,Miryam Naddaf

DOI: https://doi.org/10.1038/d41586-024-02719-5

IF: 64.8

2024-08-31

Nature

Abstract:Data from giant project show how withdrawn research propagates through the literature. Data from giant project show how withdrawn research propagates through the literature.

multidisciplinary sciences
Hidden: A Baker's Dozen Ways in Which Research Reporting is Less Transparent than it Could be and Suggestions for Implementing Einstein's Dictum

Abu Bakkar Siddique,Brian Shaw,Johanna Dwyer,David A Fields,Kevin Fontaine,David Hand,Randy Schekman,Jeffrey Alberts,Julie Locher,David B Allison

DOI: https://doi.org/10.1007/s11948-024-00517-w

2024-10-16

Abstract:The tutelage of our mentors as scientists included the analogy that writing a good scientific paper was an exercise in storytelling that omitted unessential details that did not move the story forward or that detracted from the overall message. However, the advice to not get lost in the details had an important flaw. In science, it is the many details of the data themselves and the methods used to generate and analyze them that give conclusions their probative meaning. Facts may sometimes slow or distract from the clarity, tidiness, intrigue, or flow of the narrative, but nevertheless they are important for the assessment of what was done, the trustworthiness of the science, and the meaning of the findings. Nevertheless, many critical elements and facts about research studies may be omitted from the narrative and become hidden from scholarly scrutiny. We describe a "baker's dozen" shortfalls in which such elements that are pertinent to evaluating the validity of scientific studies are sometimes hidden in reports of the work. Such shortfalls may be intentional or unintentional or lie somewhere in between. Additionally, shortfalls may occur at the level of the individual or an institution or of the entire system itself. We conclude by proposing countermeasures to these shortfalls.
The citation disadvantage of clinical research

Qing Ke

DOI: https://doi.org/10.1016/j.joi.2019.100998

IF: 3.7

2020-02-01

Journal of Informetrics

Abstract:<p>Biomedical research encompasses diverse types of activities, from basic science ("bench") to clinical medicine ("bedside") to bench-to-bedside translational research. It, however, remains unclear whether different types of research receive citations at varying rates. Here we aim to answer this question by using a newly proposed paper-level indicator that quantifies the extent to which a paper is basic science or clinical medicine. Applying this measure to 5 million biomedical papers, we find a systematic citation disadvantage of clinical oriented papers; they tend to garner far fewer citations and are less likely to be hit works than papers oriented towards basic science. At the same time, clinical research has a higher variance in its citation. We also find that the citation difference between basic and clinical research decreases, yet still persists, if longer citation-window is used. Given the increasing adoption of short-term, citation-based bibliometric indicators in funding decisions, the under-cited effect of clinical research may provide disincentives for bio-researchers to venture into the translation of basic scientific discoveries into clinical applications, thus providing explanations of reasons behind the existence of the gap between basic and clinical research that is commented as "valley of death" and the commentary of "extinction" risk of translational researchers. Our work may provide insights to policy-makers on how to evaluate different types of biomedical research.</p>

information science & library science,computer science, interdisciplinary applications
The Sci-Hub effect on papers’ citations

Juan C. Correa,Henry Laverde-Rojas,Julian Tejada,Fernando Marmolejo-Ramos

DOI: https://doi.org/10.1007/s11192-020-03806-w

IF: 3.801

2021-01-25

Scientometrics

Abstract:Citations are often used as a metric of the impact of scientific publications. Here, we examine how the number of downloads from Sci-Hub as well as various characteristics of publications and their authors predicts future citations. Using data from 12 leading journals in economics, consumer research, neuroscience, and multidisciplinary research, we found that articles downloaded from Sci-Hub were cited 1.72 times more than papers not downloaded from Sci-Hub and that the number of downloads from Sci-Hub was a robust predictor of future citations. Among other characteristics of publications, the number of figures in a manuscript consistently predicts its future citations. The results suggest that limited access to publications may limit some scientific research from achieving its full impact.

information science & library science,computer science, interdisciplinary applications
Distinct citation distributions complicate research evaluations. A single indicator that universally reveals research efficiency cannot be formulated

Alonso Rodríguez-Navarro

2024-07-12

Abstract:Purpose: Analyze the diversity of citation distributions to publications in different research topics to investigate the accuracy of size-independent, rank-based indicators. Top percentile-based indicators are the most common indicators of this type, and the evaluations of Japan are the most evident misjudgments. Design/methodology/approach: The distributions of citations to publications from countries and in journals in several research topics were analyzed along with the corresponding global publications using histograms with logarithmic binning, double rank plots, and normal probability plots of log-transformed numbers of citations. Findings: Size-independent, top percentile-based indicators are accurate when the global ranks of local publications fit a power law, but deviations in the least cited papers are frequent in countries and occur in all journals with high impact factors. In these cases, a single indicator is misleading. Comparisons of proportions of uncited papers are the best way to predict these deviations. Research limitations: The study is fundamentally analytical; its results describe mathematical facts that are self-evident. Practical implications: Respectable institutions, such as the OECD, European Commission, US National Science Board, and others, produce research country rankings and individual evaluations using size-independent percentile indicators that are misleading in many countries. These misleading evaluations should be discontinued because they cause confusion among research policymakers and lead to incorrect research policies. Originality/value: Studies linking the lower tail of citation distribution, including uncited papers, to percentile research indicators have not been performed previously. The present results demonstrate that studies of this type are necessary to find reliable procedures for research assessments.

Digital Libraries,Information Retrieval
Are All the References Mentioned in Academic Papers? A Quantitative Analysis of the Nonmentioned Phenomenon

Weibin Wang,Guang Yu,Tian Yu,Zheng Wang

DOI: https://doi.org/10.1080/00987913.2020.1856763

2021-01-01

Serials Review

Abstract:The nonmentioned phenomenon is when citations or references are missing from the body of the paper, and the phenomenon may be abnormal citing behavior that results from the negligence of the author or editor. Therefore, this study selected 40 journals in five different disciplines and carried out a comparative study. The research methods employed in this study were multivariate statistical analysis and similarity calculations. The results show that 3.45% of papers included the nonmentioned phenomenon and that the similarity between nonmentioned references and the cited paper are so low as to disqualify part of the nonmentioned references. Further, since the nonmentioned phenomenon did not appear in 96.55% of papers, it can be concluded that although most of the papers are consistent with citation normativeness, the nonmentioned phenomenon does exist in scientific papers. This study provides evidence to suggest that the nonmentioned phenomenon should receive more attention in the academic world.
Trends in Self-citation Rates in High-impact Neurology, Neuroscience, and Psychiatry Journals

Matthew Rosenblatt,Saloni Mehta,Hannah Peterson,Javid Dadashkarimi,Raimundo X Rodriguez,Maya L Foster,Brendan D Adkinson,Qinghao Liang,Violet M Kimble,Jean Ye,Marie C McCusker,Michael C Farruggia,Max Rolison,Margaret L Westwater,Rongtao Jiang,Stephanie Noble,Dustin Scheinost

DOI: https://doi.org/10.1101/2022.09.27.509533

2024-07-23

Abstract:Citation metrics influence academic reputation and career trajectories. Recent works have highlighted flaws in citation practices in the Neurosciences, such as the under-citation of women. However, self-citation rates--or how much authors cite themselves--have not yet been comprehensively investigated in the Neurosciences. This work characterizes self-citation rates in basic, translational, and clinical Neuroscience literature by collating 100,347 articles from 63 journals between the years 2000-2020. In analyzing over five million citations, we demonstrate four key findings: 1) increasing self-citation rates of Last Authors relative to First Authors, 2) lower self-citation rates in low- and middle-income countries, 3) gender differences in self-citation stemming from differences in the number of previously published papers, and 4) variations in self-citation rates by field. Our characterization of self-citation provides insight into citation practices that shape the perceived influence of authors in the Neurosciences, which in turn may impact what type of scientific research is done and who gets the opportunity to do it.

Neuroscience
Are Disruptive Papers More Likely to Impact Technology and Society?

Alex J. Yang,Xiaohui Yan,Haotian Hu,Hanlin Hu,Jia Kong,Sanhong Deng

DOI: https://doi.org/10.1002/asi.24947

2024-01-01

Journal of the Association for Information Science and Technology

Abstract:In exploring the intersection of scholarly research with technological advancement and societal impact, our analysis delves into nearly 40 million research papers spanning from 1950 to 2020 across all fields of study in science. Our scrutiny reveals an intriguing phenomenon: papers characterized by a higher CD index, often considered transformative, paradoxically exhibit a diminished propensity to influence technological and societal domains. This observation suggests a latent bias against the CD index, prompting a deeper inquiry into its implications. To unravel this trend, we introduce the concept of “disruptive citation,” a nuanced metric gauging the absolute disruptive impact of papers. Notably, papers drawing higher disruptive citations exhibit a significantly higher probability to influence both technological and societal spheres. Upon examining the heterogeneity across years and fields, we identify a bias against the CD index predominantly in the last two decades and within STEM fields. However, the positive effects of disruptive impact remain consistent across all years and fields. Our findings remain robust even when employing alternative measures of disruptive impact and controlling for total citations. By shedding light on these dynamics, our study seeks to enrich discussions regarding the recognition and role of disruptive scientific endeavors in shaping our world.
Sneaked references: Cooked reference metadata inflate citation counts

Lonni Besançon,Guillaume Cabanac,Cyril Labbé,Alexander Magazinov

DOI: https://doi.org/10.1002/asi.24896

2023-10-04

Abstract:We report evidence of an undocumented method to manipulate citation counts involving 'sneaked' references. Sneaked references are registered as metadata for scientific articles in which they do not appear. This manipulation exploits trusted relationships between various actors: publishers, the Crossref metadata registration agency, digital libraries, and bibliometric platforms. By collecting metadata from various sources, we show that extra undue references are actually sneaked in at Digital Object Identifier (DOI) registration time, resulting in artificially inflated citation counts. As a case study, focusing on three journals from a given publisher, we identified at least 9% sneaked references (5,978/65,836) mainly benefiting two authors. Despite not existing in the articles, these sneaked references exist in metadata registries and inappropriately propagate to bibliometric dashboards. Furthermore, we discovered 'lost' references: the studied bibliometric platform failed to index at least 56% (36,939/65,836) of the references listed in the HTML version of the publications. The extent of the sneaked and lost references in the global literature remains unknown and requires further investigations. Bibliometric platforms producing citation counts should identify, quantify, and correct these flaws to provide accurate data to their patrons and prevent further citation gaming.

Digital Libraries
Citations: Indicators of Quality? the Impact Fallacy

Loet Leydesdorff,Lutz Bornmann,Jordan Comins,Staša Milojević

DOI: https://doi.org/10.3389/frma.2016.00001

2016-01-01

Frontiers in Research Metrics and Analytics

Abstract:We argue that citation is a composed indicator: short-term citations can be considered as currency at the research front, whereas long-term citations can contribute to the codification of knowledge claims into concept symbols. Knowledge claims at the research front are more likely to be transitory and are therefore problematic as indicators of quality. Citation impact studies focus on short-term citation, and therefore tend to measure not epistemic quality, but involvement in current discourses in which contributions are positioned by referencing. We explore this argument using three case studies: (1) citations of the journal Soziale Welt as an example of a venue that tends not to publish papers at a research front, unlike, for example, JACS; (2) Robert Merton as a concept symbol across theories of citation; and (3) the Multi-RPYS ("Multi-Referenced Publication Year Spectroscopy") of the journals Scientometrics, Gene, and Soziale Welt. We show empirically that the measurement of "quality" in terms of citations can further be qualified: short-term citation currency at the research front can be distinguished from longer-term processes of incorporation and codification of knowledge claims into bodies of knowledge. The recently introduced Multi-RPYS can be used to distinguish between short-term and long-term impacts.
Delayed Citation Impact of Interdisciplinary Research

Yang Zhang,Yang Wang,Haifeng Du,Shlomo Havlin

DOI: https://doi.org/10.1016/j.joi.2023.101468

IF: 3.7

2024-01-01

Journal of Informetrics

Abstract:Interdisciplinary research increasingly fuels innovation, and is a key input for future breakthroughs. Yet the timing of when interdisciplinary research achieves its highest citation impact remains unclear. Here, we use the time of a paper to reach its citation peak to quantify citation dynamics, and examine its relationship with paper interdisciplinarity. Using large scale publication datasets spanning over 37 years, our results suggest that interdisciplinary papers show significant delayed citation impact both at the individual paper level and collectively, as it takes longer for highly interdisciplinary papers to reach their citation peak as well as their half citations. Such relationships are nearly universal across various scientific disciplines and time periods. Furthermore, we study the underlying forces behind this delayed impact, finding that the effect goes beyond the Matthew effect (i.e., the rich-get-richer effect). Although team size and content conventionality are partly related to the citation delay, they cannot fully explain this effect. Overall, our results suggest that governments, research administrators, and funding agencies should be aware of this general feature of interdisciplinary science, which may have broad policy implications.
Citation Data and Analysis: Limitations and Shortcomings

John L. Worrall,Ellen G. Cohn

DOI: https://doi.org/10.1177/10439862231170972

2023-05-06

Journal of Contemporary Criminal Justice

Abstract:Journal of Contemporary Criminal Justice, Ahead of Print. As a means for measuring scholarly influence, citation analysis has several limitations and shortcomings. We first review the main sources of citation data (Web of Science, Scopus, Google Scholar, and information collected directly from reference lists) and discuss the shortcomings of each source. Next, we review five significant limitations of citation analysis as a methodology (academic over popular interest, various motivations for citing, manipulation potential, failure to account for author ordering, and citations only appearing in "indexed" journals). The issues we touch on set the stage for the remainder of the articles in this special issue.

criminology & penology
Beyond Citations: Measuring Novel Scientific Ideas and their Impact in Publication Text

Sam Arts,Nicola Melluso,Reinhilde Veugelers

2024-10-04

Abstract:New scientific ideas drive progress, yet measuring them remains challenging. We use natural language processing to detect the origin and impact of new ideas in scientific publications. To validate our methods, we analyze Nobel Prize-winning papers, which likely pioneered impactful new ideas, and literature review papers, which typically consolidate existing knowledge. We also show that novel papers have more intellectual neighbors published after them, indicating they are ahead of their intellectual peers. Finally, papers introducing new ideas, particularly those with greater impact, attract more citations. Data and code are openly available.

General Economics
A Review on Citation Amnesia in Depression and Inflammation Research.

Michael Maes

2015-01-01

Neuroendocrinology letters

Abstract:Once original scientific results are published the author has the "intellectual property" and may claim ownership. Discovery credit is one of the most important "rewards" for scientists and thus incorrect credits undermine the reward system of science. Scientists who publish should therefore give proper credit and acknowledge the primary sources. Failure to do so is regarded as "citation negligence", "the disregard syndrome", "citation amnesia", "plagiarism by omission", "bibliographic plagiarism" or "citation plagiarism", and may range from an unconscious or conscious "failure to credit a prior discoverer so as to give an improper impression of priority" to "the appropriation of another person's ideas or results without given proper credit". False discovery credit is considered to be "a menace to honest science", "a serious transgression" or "intellectual theft, be it intentional or not". This paper describes some examples of citation amnesia showing that scientists often fail to credit prior sources and give false discovery credit to other scientists. One example is the association between major depression and activated immuno-inflammatory pathways, a discovery by European groups and published in many papers since 1990. Now, 25 years later, it is commonplace that these theories are credited to secondary American sources whose work in "the last decade", did or did not examine these pathways in major depression. This gives an improper impression of priority of American-based scientists. Here it is proposed that this citation amnesia and plagiarism reinforced the wrong science and had negative effects on the development of immune-inflammatory biomarkers and new immune-related treatments for depression. It is concluded that journal editors should improve their citation standards to guarantee correct assignment of discovery credit for example by demanding a signed pledge from the authors that correct citations to the primary sources were made.
Status drives how we cite: Evidence from thousands of authors

Misha Teplitskiy,Eamon Duede,Michael Menietti,Karim R. Lakhani

DOI: https://doi.org/10.48550/arXiv.2002.10033

2020-02-24

Social and Information Networks

Abstract:Researchers cite works for a variety of reasons, including some having nothing to do with acknowledging influence. The distribution of different citation types in the literature, and which papers attract which types, is poorly understood. We investigate high-influence and low-influence citations and the mechanisms producing them using 17,154 ground-truth citation types provided via survey by 9,380 authors systematically sampled across academic fields. Overall, 54% of citations denote little-to-no influence and these citations are concentrated among low status (lightly cited) papers. In contrast, high-influence citations are concentrated among high status (highly cited) papers through a number of steps that resemble a pipeline. Authors discover highly cited papers earlier in their projects, more often through social contacts, and read them more closely. Papers' status, above and beyond any quality differences, directly helps determine their pipeline: experimentally revealing or hiding citation counts during the survey shows that low counts cause lowered perceptions of quality. Accounting for citation types thus reveals a "double status effect": in addition to affecting how often a work is cited, status affects how meaningfully it is cited. Consequently, highly cited papers are even more influential than their raw citation counts suggest.

‘Hidden’ citations conceal the true impact of research

Hidden Citations Obscure True Impact in Science

Uncited articles and their effect on the concentration of citations

Citing for High Impact

Why There Are So Many Contradicted or Exaggerated Findings in Highly-Cited Clinical Research?

Some Insights into the Factors Influencing Continuous Citation of Retracted Scientific Papers

Exclusive: the papers that most heavily cite retracted studies

Hidden: A Baker's Dozen Ways in Which Research Reporting is Less Transparent than it Could be and Suggestions for Implementing Einstein's Dictum

The citation disadvantage of clinical research

The Sci-Hub effect on papers’ citations

Distinct citation distributions complicate research evaluations. A single indicator that universally reveals research efficiency cannot be formulated

Are All the References Mentioned in Academic Papers? A Quantitative Analysis of the Nonmentioned Phenomenon

Trends in Self-citation Rates in High-impact Neurology, Neuroscience, and Psychiatry Journals

Are Disruptive Papers More Likely to Impact Technology and Society?

Sneaked references: Cooked reference metadata inflate citation counts

Citations: Indicators of Quality? the Impact Fallacy

Delayed Citation Impact of Interdisciplinary Research

Citation Data and Analysis: Limitations and Shortcomings

Beyond Citations: Measuring Novel Scientific Ideas and their Impact in Publication Text

A Review on Citation Amnesia in Depression and Inflammation Research.

Status drives how we cite: Evidence from thousands of authors