Predicting Award Winning Research Papers at Publication Time

Riccardo Vella,Andrea Vitaletti,Fabrizio Silvestri

2024-06-18

Abstract:In recent years, many studies have been focusing on predicting the scientific impact of research papers. Most of these predictions are based on citations count or rely on features obtainable only from already published papers. In this study, we predict the likelihood for a research paper of winning an award only relying on information available at publication time. For each paper, we build the citation subgraph induced from its bibliography. We initially consider some features of this subgraph, such as the density and the global clustering coefficient, to make our prediction. Then, we mix this information with textual features, extracted from the abstract and the title, to obtain a more accurate final prediction. We made our experiments considering the ArnetMiner citation graph, while the ground truth on award-winning papers has been obtained from a collection of best paper awards from 32 computer science conferences. In our experiment, we obtained an encouraging F1 score of 0.694. Remarkably, The high recall and the low false negatives rate, show how the model performs very well at identifying papers that will not win an award. This behavior can help researchers in getting a first evaluation of their work at publication time. Lastly, we made some first experiments on interpretability. Our results highlight some interesting patterns both in topological and textual features.

Information Retrieval

What problem does this paper attempt to address?

The paper aims to address the issue of predicting the future award potential of research papers at the time of their publication. Specifically:

1. **Limitations of Existing Methods**: Most existing studies rely on citation counts or features that can only be obtained from already published papers to predict scientific impact, making it difficult for researchers to timely assess the quality of their work during the writing stage.
2. **Introduction of a New Method**: The paper proposes a new method that uses only the information available at the time of publication (such as the topological features of the citation subgraph and textual features) to predict whether a paper will win an award.
3. **Experimental Results**: By constructing a citation subgraph and combining it with textual features for hybrid modeling, the experimental results show that the method can effectively identify papers that will not win awards, but there is some uncertainty in predicting papers that will actually win awards. The final model achieved an F1 score of 0.694.
4. **Interpretative Experiments**: The authors also conducted preliminary interpretative experiments and found that award-winning papers have lower textual similarity with the articles they cite, and these articles are more sparsely distributed in the network.

Overall, this study provides researchers with a tool to assess the potential impact of their papers before publication.