A dispersion-driven method for grant and proposal allocation

N. Mirabal,J. L. Contreras

DOI: https://doi.org/10.48550/arXiv.1210.5450

2012-10-19

Abstract:Inspired by a recent editorial (Langer 2012), we suggest a relative simple scheme to grade grants and proposals that might enhance scientific innovation.

Instrumentation and Methods for Astrophysics,Physics and Society

What problem does this paper attempt to address?

Contest models highlight inherent inefficiencies of scientific funding competitions

Kevin Gross,Carl T. Bergstrom

DOI: https://doi.org/10.1371/journal.pbio.3000065

IF: 9.8

2019-01-02

PLoS Biology

Abstract:Scientific research funding is allocated largely through a system of soliciting and ranking competitive grant proposals. In these competitions, the proposals themselves are not the deliverables that the funder seeks, but instead are used by the funder to screen for the most promising research ideas. Consequently, some of the funding program's impact on science is squandered because applying researchers must spend time writing proposals instead of doing science. To what extent does the community's aggregate investment in proposal preparation negate the scientific impact of the funding program? Are there alternative mechanisms for awarding funds that advance science more efficiently? We use the economic theory of contests to analyze how efficiently grant proposal competitions advance science, and compare them with recently proposed, partially randomized alternatives such as lotteries. We find that the effort researchers waste in writing proposals may be comparable to the total scientific value of the research that the funding supports, especially when only a few proposals can be funded. Moreover, when professional pressures motivate investigators to seek funding for reasons that extend beyond the value of the proposed science (e.g., promotion, prestige), the entire program can actually hamper scientific progress when the number of awards is small. We suggest that lost efficiency may be restored either by partial lotteries for funding or by funding researchers based on past scientific success instead of proposals for future work.

biochemistry & molecular biology,biology
Theoretical research without projects

Miguel Navascues,Costantino Budroni

DOI: https://doi.org/10.1371/journal.pone.0214026

2019-07-05

Abstract:We propose a funding scheme for theoretical research that does not rely on project proposals, but on recent past scientific productivity. Given a quantitative figure of merit on the latter and the total research budget, we introduce a number of policies to decide the allocation of funds in each grant call. Under some assumptions on scientific productivity, some of such policies are shown to converge, in the limit of many grant calls, to a funding configuration that is close to the maximum total productivity of the whole scientific community. We present numerical simulations showing evidence that these schemes would also perform well in the presence of statistical noise in the scientific productivity and/or its evaluation. Finally, we prove that one of our policies cannot be cheated by individual research units. Our work must be understood as a first step towards a mathematical theory of the research activity.

Optimization and Control,Physics and Society
Flipping the grant application review process

Ivo D Dinov

DOI: https://doi.org/10.1080/03075079.2019.1628201

Abstract:The return on research investment resulting from new breakthrough scientific discoveries may be decreasing over time due to the law of diminishing returns, the relative decrease of research funding in terms of purchasing power parity, and various activities gaming the system. By altering the grant-review process, the scientific community may directly address the third problem. There is evidence that peer reviews of research proposals may lack reliability and may produce invalid or inconsistent ratings. In addition, extreme focus on grantsmanship threatens to uproot a cornerstone principle that scientific-value should be the key driver in funding decision-making. This opinion provides (1) a justification of the need to consider alternative strategies to boost the impact of public investment in innovative scientific discovery, (2) proposes a framework for flipping the traditional front-loaded peer-review approach to allocation of research funding, into a new back-loaded assessment of scholarly return on investment, and (3) provokes the scientific community to accelerate the debate on alternative funding mechanisms, as the stakes of inaction may be very high.
Rethinking Resource Allocation in Science

Johan Bollen,Stephen Carpenter,Jane Lubchenco,Marten Scheffer

DOI: https://doi.org/10.48550/arXiv.1904.07645

2019-04-16

Computers and Society

Abstract:US funding agencies alone distribute a yearly total of roughly $65B dollars largely through the process of proposal peer review: scientists compete for project funding by submitting grant proposals which are evaluated by selected panels of peer reviewers. Similar funding systems are in place in most advanced democracies. However, in spite of its venerable history, proposal peer review is increasingly struggling to deal with the increasing mismatch between demand and supply of research funding.
Collective allocation of science funding: from funding agencies to scientific agency

Johan Bollen,David Crandall,Damion Junk,Ying Ding,Katy Boerner

DOI: https://doi.org/10.48550/arXiv.1304.1067

2013-04-04

Abstract:Public agencies like the U.S. National Science Foundation (NSF) and the National Institutes of Health (NIH) award tens of billions of dollars in annual science funding. How can this money be distributed as efficiently as possible to best promote scientific innovation and productivity? The present system relies primarily on peer review of project proposals. In 2010 alone, NSF convened more than 15,000 scientists to review 55,542 proposals. Although considered the scientific gold standard, peer review requires significant overhead costs, and may be subject to biases, inconsistencies, and oversights. We investigate a class of funding models in which all participants receive an equal portion of yearly funding, but are then required to anonymously donate a fraction of their funding to peers. The funding thus flows from one participant to the next, each acting as if he or she were a funding agency themselves. Here we show through a simulation conducted over large-scale citation data (37M articles, 770M citations) that such a distributed system for science may yield funding patterns similar to existing NIH and NSF distributions, but may do so at much lower overhead while exhibiting a range of other desirable features. Self-correcting mechanisms in scientific peer evaluation can yield an efficient and fair distribution of funding. The proposed model can be applied in many situations in which top-down or bottom-up allocation of public resources is either impractical or undesirable, e.g. public investments, distribution chains, and shared resource management.

Physics and Society,Digital Libraries
The choice: evaluating and selecting scientific proposals

Antonio Navarra

DOI: https://doi.org/10.48550/arXiv.1504.05315

2015-04-21

Abstract:The selection process of proposals is a crucial component of scientific progress and innovations. Limited resources must be allocated in the most effective way to maximise advancements and the production of new knowledge, especially as it is becoming increasingly clear that technological and scientific innovation and creativity is an instrument of economic policy and social development. The traditional approach based on merit evaluation by experts has been the preferred method, but there is an issue regarding to what extent such a method can also be an instrument of effective policy. This paper discuss some of the basic processes involved in the evaluation and selection of proposals, indicating some criterion for an optimal solution.

Physics and Society
A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications

Lutz Bornmann,Loet Leydesdorff,Peter Van den Besselaar

DOI: https://doi.org/10.1016/j.joi.2009.10.004

IF: 3.7

2010-01-01

Journal of Informetrics

Abstract:Combining different data sets with information on grant and fellowship applications submitted to two renowned funding agencies, we are able to compare their funding decisions (award and rejection) with scientometric performance indicators across two fields of science (life sciences and social sciences). The data sets involve 671 applications in social sciences and 668 applications in life sciences. In both fields, awarded applicants perform on average better than all rejected applicants. If only the most preeminent rejected applicants are considered in both fields, they score better than the awardees on citation impact. With regard to productivity we find differences between the fields. While the awardees in life sciences outperform on average the most preeminent rejected applicants, the situation is reversed in social sciences.
Impact of Declining Proposal Success Rates on Scientific Productivity

Priscilla Cushman,J. Todd Hoeksema,Chryssa Kouveliotou,James Lowenthal,Bradley Peterson,Keivan G. Stassun,Ted von Hippel

DOI: https://doi.org/10.48550/arXiv.1510.01647

2015-10-10

Abstract:Over the last decade proposal success rates in the fundamental sciences have dropped significantly. Astronomy and related fields funded by NASA and NSF are no exception. Data across agencies show that this is not principally the result of a decline in proposal merit (the proportion of proposals receiving high rankings is largely unchanged), nor of a shift in proposer demographics (seniority, gender, and institutional affiliation have all remained unchanged), nor of an increase (beyond inflation) in the average requested funding per proposal, nor of an increase in the number of proposals per investigator in any one year. Rather, the statistics are consistent with a scenario in which agency budgets for competed research are flat or decreasing in inflation-adjusted dollars, the overall population of investigators has grown, and a larger proportion of these investigators are resubmitting meritorious but unfunded proposals. This White Paper presents statistics which support this conclusion, as well as recent research on the time cost of proposal writing versus that of producing publishable results. We conclude that an aspirational proposal success rate of 30-35% would still provide a healthily competitive environment for researchers, would more fully utilize the scientific capacity of the community's facilities and missions, and provide relief to the funding agencies who face the logistics of ever-increasing volumes of proposals.

Physics and Society,Cosmology and Nongalactic Astrophysics,Earth and Planetary Astrophysics,Astrophysics of Galaxies,High Energy Astrophysical Phenomena,Solar and Stellar Astrophysics
Incentives, Quality, and Risks: A Look Into the NSF Proposal Review Pilot

Parinaz Naghizadeh,Mingyan Liu

DOI: https://doi.org/10.48550/arXiv.1307.6528

2013-07-25

Abstract:The National Science Foundation (NSF) will be experimenting with a new distributed approach to reviewing proposals, whereby a group of principal investigators (PIs) or proposers in a subfield act as reviewers for the proposals submitted by the same set of PIs. To encourage honesty, PIs' chances for getting funded are tied to the quality of their reviews (with respect to the reviews provided by the entire group), in addition to the quality of their proposals. Intuitively, this approach can more fairly distribute the review workload, discourage frivolous proposal submission, and encourage high quality reviews. On the other hand, this method has already raised concerns about the integrity of the process and the possibility of strategic manipulation. In this paper, we take a closer look at three specific issues in an attempt to gain a better understanding of the strengths and limitations of the new process beyond first impressions and anecdotal evidence. We start by considering the benefits and drawbacks of bundling the quality of PIs' reviews with the scientific merit of their proposals. We then consider the issue of collusion and favoritism. Finally, we examine whether the new process puts controversial proposals at a disadvantage. We conclude that some benefits of using review quality as an incentive mechanism may outweigh its drawbacks. On the other hand, even a coalition of two PIs can cause significant harm to the process, as the built-in incentives are not strong enough to deter collusion. While we also confirm the common suspicion that the process is skewed toward non-controversial proposals, the more unexpected finding is that among equally controversial proposals, those of lower quality get a leg up through this process. Thus the process not only favors non-controversial proposals, but in some sense, mediocrity. We also discuss possible ways to improve this review process.

Computer Science and Game Theory,Social and Information Networks,Physics and Society
Designing Scientific Grants

Christoph Carnehl,Marco Ottaviani,Justus Preusser

2024-10-16

Abstract:This paper overviews the economics of scientific grants, focusing on the interplay between the inherent uncertainty in research, researchers' incentives, and grant design. Grants differ from traditional market systems and other science and innovation policy tools, such as prizes and patents. We outline the main economic forces specific to science, noting the limited attention given to grant funding in the economics literature. Using tools from information economics, we identify key incentive problems at various stages of the grant funding process and offer guidance for effective grant design. In the allocation stage, funders aim to select the highest-merit applications while minimizing evaluation costs. The selection rule, in turn, impacts researchers' incentives to apply and invest in their proposals. In the grant management stage, funders monitor researchers to ensure efficient use of funds. We discuss the advantages and potential pitfalls of (partial) lotteries and emphasize the effectiveness of staged grant design in promoting a productive use of grants. Beyond these broadly applicable insights, our overview highlights the need for further research on grantmaking. Understudied areas include, at the micro level, the interplay of different grant funding stages, and at the macro level, the interaction of grants with other instruments in the market for science.

General Economics
A decision support approach for assigning reviewers to proposals

Yunhong Xu,Jian Ma,Yonghong Sun,Gang Hao,Wei Xu,Dingtao Zhao

DOI: https://doi.org/10.1016/j.eswa.2010.03.027

IF: 8.5

2010-01-01

Expert Systems with Applications

Abstract:Peer review plays an important role in research project selection at funding agencies. Quality of peer review greatly depends on the degree of matching between reviewers and assigned research proposals. This kind of matching is largely determined by the process of assigning proposals to reviewers. As the number of proposals submitted to funding agencies continues to grow, the traditional approaches of finding appropriate reviewers for each individual proposal fail to satisfy the practical needs. This paper proposes an alternative approach whose basic idea is grouping proposals first and then assigning appropriate reviewers for each proposal group. Based on the idea, a decision support approach is proposed to identify valid proposals and reviewers, classify proposals and reviewers according to their disciplines, partition proposals into groups and assign reviewers to proposal groups. A system has been developed based on the proposed approach to facilitate the decision making process of assigning proposals to reviewers.
Promotional Language and the Adoption of Innovative Ideas in Science

Hao Peng,Huilian Sophie Qiu,Henrik Barslund Fosse,Brian Uzzi

2024-06-08

Abstract:How are the merits of innovative ideas communicated in science? Here we conduct semantic analyses of grant application success with a focus on scientific promotional language, which has been growing in frequency in many contexts and purportedly may convey an innovative idea's originality and significance. Our analysis attempts to surmount limitations of prior studies by examining the full text of tens of thousands of both funded and unfunded grants from three leading public and private funding agencies: the NIH, the NSF, and the Novo Nordisk Foundation, one of the world's largest private science foundations. We find a robust association between promotional language and the support and adoption of innovative ideas by funders and other scientists. First, the percentage of promotional language in a grant proposal is associated with up to a doubling of the grant's probability of being funded. Second, a grant's promotional language reflects its intrinsic level of innovativeness. Third, the percentage of promotional language predicts the expected citation and productivity impact of publications that are supported by funded grants. Lastly, a computer-assisted experiment that manipulates the promotional language in our data demonstrates how promotional language can communicate the merit of ideas through cognitive activation. With the incidence of promotional language in science steeply rising, and the pivotal role of grants in converting promising and aspirational ideas into solutions, our analysis provides empirical evidence that promotional language is associated with effectively communicating the merits of innovative scientific ideas.

Digital Libraries,Computation and Language,Computers and Society
GotFunding: A grant recommendation system based on scientific articles

Tong Zeng,Daniel E. Acuna

DOI: https://doi.org/10.1002/pra2.323

2024-05-21

Abstract:Obtaining funding is an important part of becoming a successful scientist. Junior faculty spend a great deal of time finding the right agencies and programs that best match their research profile. But what are the factors that influence the best publication--grant matching? Some universities might employ pre-award personnel to understand these factors, but not all institutions can afford to hire them. Historical records of publications funded by grants can help us understand the matching process and also help us develop recommendation systems to automate it. In this work, we present \textsc{GotFunding} (Grant recOmmendaTion based on past FUNDING), a recommendation system trained on National Institutes of Health's (NIH) grant--publication records. Our system achieves a high performance (NDCG@1 = 0.945) by casting the problem as learning to rank. By analyzing the features that make predictions effective, our results show that the ranking considers most important 1) the year difference between publication and grant grant, 2) the amount of information provided in the publication, and 3) the relevance of the publication to the grant. We discuss future improvements of the system and an online tool for scientists to try.

Information Retrieval,Digital Libraries,Machine Learning
Enhancing Peer Review in Astronomy: A Machine Learning and Optimization Approach to Reviewer Assignments for ALMA

John M. Carpenter,Andrea Corvillón,Nihar B. Shah

2024-10-14

Abstract:The increasing volume of papers and proposals undergoing peer review emphasizes the pressing need for greater automation to effectively manage the growing scale. In this study, we present the deployment and evaluation of machine learning and optimization techniques for assigning proposals to reviewers that was developed for the Atacama Large Millimeter/submillimeter Array (ALMA) during the Cycle 10 Call for Proposals issued in 2023. By utilizing topic modeling algorithms, we identify the proposal topics and assess reviewers' expertise based on their historical ALMA proposal submissions. We then apply an adapted version of the assignment optimization algorithm from PeerReview4All (Stelmakh et al. 2021a) to maximize the alignment between proposal topics and reviewer expertise. Our evaluation shows a significant improvement in matching reviewer expertise: the median similarity score between the proposal topic and reviewer expertise increased by 51 percentage points compared to the previous cycle, and the percentage of reviewers reporting expertise in their assigned proposals rose by 20 percentage points. Furthermore, the assignment process proved highly effective in that no proposals required reassignment due to significant mismatches, resulting in a savings of 3 to 5 days of manual effort.

Instrumentation and Methods for Astrophysics,Artificial Intelligence,Digital Libraries
Toward predicting research proposal success

Kevin W. Boyack,Caleb Smith,Richard Klavans

DOI: https://doi.org/10.1007/s11192-017-2609-2

IF: 3.801

2017-12-05

Scientometrics

Abstract:Citation analysis and discourse analysis of 369 R01 NIH proposals are used to discover possible predictors of proposal success. We focused on two issues: the Matthew effect in science—Merton’s claim that eminent scientists have an inherent advantage in the competition for funds—and quality of writing or clarity. Our results suggest that a clearly articulated proposal is more likely to be funded than a proposal with lower quality of discourse. We also find that proposal success is correlated with a high level of topical overlap between the proposal references and the applicant’s prior publications. Implications associated with the analysis of proposal data are discussed.

information science & library science,computer science, interdisciplinary applications
Understanding the assembly of interdisciplinary teams and its impact on performance

Alina Lungeanu,Yun Huang,Noshir S. Contractor

DOI: https://doi.org/10.1016/j.joi.2013.10.006

IF: 3.7

2014-01-01

Journal of Informetrics

Abstract:Interdisciplinary teams are assembled in scientific research and are aimed at solving complex problems. Given their increasing importance, it is not surprising that considerable attention has been focused on processes of collaboration in interdisciplinary teams. Despite such efforts, we know less about the factors affecting the assembly of such teams in the first place. In this paper, we investigate the structure and the success of interdisciplinary scientific research teams. We examine the assembly factors using a sample of 1,103 grant proposals submitted to two National Science Foundation interdisciplinary initiatives during a 3-year period, including both awarded and non-awarded proposals. The results indicate that individuals' likelihood of collaboration on a proposal is higher among those with longer tenure, lower institutional tier, lower H-index, and with higher levels of prior co-authorship and citation relationships. However, successful proposals have a little bit different relational patterns: individuals' likelihood of collaboration is higher among those with lower institutional tier, lower H-index, (female) gender, higher levels of prior co-authorship, but with lower levels of prior citation relationships.

information science & library science,computer science, interdisciplinary applications
Competition for funding as an indicator of research competitiveness

Clara Eugenia Garcia,Luis Sanz-Menéndez

DOI: https://doi.org/10.1007/s11192-005-0251-x

IF: 3.801

2005-08-01

Scientometrics

Abstract:Research quality is the cornerstone of modern science, it is used in the understanding of reputational differences among scientific and academic institutions. Traditionally, scientific activity is measured by a set of indicators and well-established bibliometric techniques based on the number of academic papers published in top-ranked journals or on the number of citations of these papers. These indicators are usually critical in measuring differences in research performance, both at individual and at scientific institutional levels. In this paper, we introduce an alternative and complementary set of indicators based on the results of competition for research funding, that aims to enlarge the framework in which research performance has traditionally been measured. Theoretical support for this paper is found in the role that the search for funding plays in the researchers’ credibility cycle as well as in peer review, the basic instrument for the allocation of public R&D funds. Our method analyses the outcomes of the researchers’ struggle for funding, using data from research proposal applications and awards, as the unit of observation, and aggregating them by research institutions to rank them in relative scales of research competitiveness.

information science & library science,computer science, interdisciplinary applications
A Network Perspective on Success in Collaboration: Stop Citing Me for Our Own Good?

Alina Lungeanu,Yun Huang,Noshir Contractor

DOI: https://doi.org/10.5465/ambpp.2012.16989abstract

2012-01-01

Academy of Management Proceedings

Abstract:Interdisciplinary teams are established in scientific research with the goal of solving complex problems. Given their increasing importance, it is not surprising that there has been considerable attention focused on the processes by which interdisciplinary teams collaborate after they are assembled. However, there has been much less research exploring the factors affecting the assembly of such teams. In this paper, we investigate the formation and the success of interdisciplinary scientific research teams. We test our hypotheses on a sample of 1,103 successful and unsuccessful grant proposals submitted to two NSF interdisciplinary programs during a 3-year period. Our results indicate that individuals’ likelihood of collaborating on a proposal is higher among those with longer tenure, lower institutional tier, lower H-index, along with higher levels of prior co-authorship and citation relationships. Our results also indicate individuals’ likelihood of collaborating on a successful proposal is higher among those with longer tenure, (female) gender, lower institutional tier, lower H-index, higher levels of prior co-authorship, but lower levels of prior citation relationships.
Promoting and Supporting Innovation Through Collaborative Research

Jim Gentile

DOI: https://doi.org/10.1096/fasebj.26.1_supplement.617.1

2012-04-01

The FASEB Journal

Abstract:Many of the major scientific challenges of the 21st century are likely to require collaborative and interdisciplinary teams. Creating good “team players” who can work synergistically to undertake high‐risk, high‐reward research should be a priority, and demonstrating success in collaborations should be encouraged by private and federal granting agencies, and more highly rewarded by universities in tenure and promotion decisions. Given the challenges that our nation faces, tenure and promotion decisions should no longer be based merely on the number of papers published or overhead dollars generated. Journal editorial boards and foundation peer‐review panels are essential to the highest‐quality science, but they should not be viewed as surrogate members of tenure and promotion committees. Rewarding careers paths should be clearly visible to the nation's newly minted science Ph.D.s, postdoctoral researchers and “tenure track” professors. Allowing these early‐career scientists more expansive opportunities during their most creative years would make the United States, and the academic communities in which they serve, an increasingly more vibrant and creative environments. The heart and soul of American science and enterprise is the willingness to take risks, to learn from failure and to repeat the process until success is achieved. Our early‐career scientists should have the same opportunity.

cell biology,biochemistry & molecular biology,biology
Credit allocation based on journal impact factor and coauthorship contribution

Javier E. Contreras-Reyes

DOI: https://doi.org/10.48550/arXiv.1606.04139

2016-05-08

Digital Libraries

Abstract:Some research institutions demand researchers to distribute the incomes they earn from publishing papers to their researchers and/or co-authors. In this study, we deal with the Impact Factor-based ranking journal as a criteria for the correct distribution of these incomes. We also include the Authorship Credit factor for distribution of the incomes among authors, using the geometric progression of Cantor's theory and the Harmonic Credit Index. Depending on the ranking of the journal, the proposed model develops a proper publication credit allocation among all authors. Moreover, our tool can be deployed in the evaluation of an institution for a funding program, as well as calculating the amounts necessary to incentivize research among personnel.

A dispersion-driven method for grant and proposal allocation

Contest models highlight inherent inefficiencies of scientific funding competitions

Theoretical research without projects

Flipping the grant application review process

Rethinking Resource Allocation in Science

Collective allocation of science funding: from funding agencies to scientific agency

The choice: evaluating and selecting scientific proposals

A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications

Impact of Declining Proposal Success Rates on Scientific Productivity

Incentives, Quality, and Risks: A Look Into the NSF Proposal Review Pilot

Designing Scientific Grants

A decision support approach for assigning reviewers to proposals

Promotional Language and the Adoption of Innovative Ideas in Science

GotFunding: A grant recommendation system based on scientific articles

Enhancing Peer Review in Astronomy: A Machine Learning and Optimization Approach to Reviewer Assignments for ALMA

Toward predicting research proposal success

Understanding the assembly of interdisciplinary teams and its impact on performance

Competition for funding as an indicator of research competitiveness

A Network Perspective on Success in Collaboration: Stop Citing Me for Our Own Good?

Promoting and Supporting Innovation Through Collaborative Research

Credit allocation based on journal impact factor and coauthorship contribution