Abstract:Critique between peers plays a vital role in the production of scientific knowledge. Yet, there is limited empirical evidence on the origins of criticism, its effects on the papers and individuals involved, and its visibility within the scientific literature. Here, we address these gaps through a data-driven analysis of papers that received substantiated and explicit written criticisms. Our analysis draws on data representing over 3,000 ``critical letters'' -- papers explicitly published to critique another -- from four high profile journals, with each letter linked to its target paper. We find that the papers receiving critical letters are disproportionately among the most highly-cited in their respective journal and, to a lesser extent, among the most interdisciplinary and novel. However, despite the theoretical importance of criticism in scientific progress, we observe no evidence that receiving a critical letter affects a paper's citation trajectory or the productivity and citation impact of its authors. One explanation for the limited consequence of critical letters is that they often go unnoticed. Indeed, we find that critical letters attract only a small fraction of the citations received by their targets, even years after publication. An analysis of topical similarity between criticized papers and their citing papers indicates that critical letters are primarily cited by researchers actively engaged in a similar field of study, whereas they are overlooked by more distant communities. Although criticism is celebrated as a cornerstone to science, our findings reveal that it is concentrated on high-impact papers, has minimal measurable consequences, and suffers from limited visibility. These results raise important questions about the role and value of critique in scientific practice.
What problem does this paper attempt to address?
This paper attempts to solve the following key problems:
1. **Which papers will receive criticism?**
- The authors explored the differences in characteristics between papers that received criticism and those that did not. Specifically, they studied factors such as the attention (number of citations) of the papers, their inter - disciplinarity, and their novelty.
- The study found that papers that received criticism were often highly - cited papers in journals and had a high level of attention. In addition, these papers were also more likely to be interdisciplinary and novel, although this association was relatively weak and was often influenced by the number of citations.
2. **What are the impacts of receiving criticism on the papers and their authors?**
- Regarding the impact of criticism on the number of citations of papers, there are several hypotheses: criticism may reduce the number of citations (because of the identification of flaws in methods or explanations), increase the number of citations (because of increased visibility), or have no impact at all (because the criticism was not widely read or was considered irrelevant).
- The research results showed that criticism letters had almost no significant impact on the citation trajectories of papers. In most journals, criticism letters did not significantly change the citation growth of papers. The only exception was in the "Other APS" journal, where criticized papers might have a slightly increased number of citations, but this might be statistical noise.
3. **What is the visibility and spread range of criticism letters?**
- Although criticism is considered crucial in scientific progress, the study found that criticism letters usually attracted only a small fraction of the citations of their target papers, even after many years.
- Criticism letters were mainly cited by researchers in the same field, while researchers in other fields paid less attention to these criticism letters. This indicates that the visibility of criticism letters is limited, especially in communities far from the field of the original paper.
### Specific Findings
- **High Attention**: Papers that received criticism were usually highly - cited papers, especially in top multidisciplinary journals such as Nature, Science, and the Proceedings of the National Academy of Sciences (PNAS). For example, in PNAS, the average citation percentile rank of criticized papers was 67.7%, while in PRL it was 54.2%.
- **Inter - disciplinarity**: Criticized papers were also more likely to be interdisciplinary, but this association was weak. For example, in PNAS, the average percentile rank of reference diversity of criticized papers was 56.4%, while in PRL it was 51.4%.
- **Novelty**: Novel papers were also more likely to receive criticism, but this association was also often influenced by the number of citations. For example, in PRL, the average novelty percentile rank of criticized papers was 55.2%.
- **Author Characteristics**: The demographic characteristics of authors (such as gender, seniority, and institutional prestige) had a relatively small impact on whether they received criticism. First authors with higher seniority were more likely to receive criticism, but there was no significant difference among last authors.
### Conclusion
This paper reveals the origin, consequences, and visibility of scientific criticism through data analysis. Although criticism is theoretically crucial for scientific progress, practical research shows that criticism letters have almost no significant impact on the citation trajectories of papers and the career development of authors, and the visibility of criticism letters is limited. These results raise important questions about the role and value of criticism in scientific practice.