Cash for catching scientific errors

Julian Nowogrodzki

DOI: https://doi.org/10.1038/d41586-024-02681-2

IF: 64.8

2024-08-20

Nature

Abstract:The ERROR project offers researchers a bounty for spotting mistakes in published papers — a strategy borrowed from the software industry. The ERROR project offers researchers a bounty for spotting mistakes in published papers — a strategy borrowed from the software industry.

multidisciplinary sciences

What problem does this paper attempt to address?

Pay researchers to spot errors in published papers

Malte Elson

DOI: https://doi.org/10.1038/d41586-024-01465-y

IF: 64.8

2024-05-22

Nature

Abstract:Borrowing the idea of 'bug bounties' from the technology industry could provide a systematic way to detect and correct the errors that litter the scientific literature. Borrowing the idea of 'bug bounties' from the technology industry could provide a systematic way to detect and correct the errors that litter the scientific literature.

multidisciplinary sciences
Facing the Facts

Patrick O'Beirne

DOI: https://doi.org/10.48550/arXiv.0803.3394

2008-03-24

Abstract:Human error research on overconfidence supports the benefits of early visibility of defects and disciplined development. If risk to the enterprise is to be reduced, individuals need to become aware of the reality of the quality of their work. Several cycles of inspection and defect removal are inevitable. Software Quality Management measurements of defect density and removal efficiency are applicable. Research of actual spreadsheet error rates shows data consistent with other software depending on the extent to which the work product was reviewed before inspection. The paper argues that the payback for an investment in early review time is justified by the saving in project delay and expensive errors in use. 'If debugging is the process of removing bugs, then programming must be the process of putting them in' - Anon.

Human-Computer Interaction
An Empirical Study of End-User Behaviour in Spreadsheet Error Detection & Correction

Brian Bishop,Kevin McDaid

DOI: https://doi.org/10.48550/arXiv.0802.3479

2008-02-24

Abstract:Very little is known about the process by which end-user developers detect and correct spreadsheet errors. Any research pertaining to the development of spreadsheet testing methodologies or auditing tools would benefit from information on how end-users perform the debugging process in practice. Thirteen industry-based professionals and thirty-four accounting & finance students took part in a current ongoing experiment designed to record and analyse end-user behaviour in spreadsheet error detection and correction. Professionals significantly outperformed students in correcting certain error types. Time-based cell activity analysis showed that a strong correlation exists between the percentage of cells inspected and the number of errors corrected. The cell activity data was gathered through a purpose written VBA Excel plug-in that records the time and detail of all cell selection and cell change actions of individuals.

Human-Computer Interaction,Computers and Society
Artificial Bugs for Crowdsearch

Hans Gersbach,Fikri Pitsuwan,Pio Blieske

DOI: https://doi.org/10.48550/arXiv.2403.09484

2024-03-14

Abstract:Bug bounty programs, where external agents are invited to search and report vulnerabilities (bugs) in exchange for rewards (bounty), have become a major tool for companies to improve their systems. We suggest augmenting such programs by inserting artificial bugs to increase the incentives to search for real (organic) bugs. Using a model of crowdsearch, we identify the efficiency gains by artificial bugs, and we show that for this, it is sufficient to insert only one artificial bug. Artificial bugs are particularly beneficial, for instance, if the designer places high valuations on finding organic bugs or if the budget for bounty is not sufficiently high. We discuss how to implement artificial bugs and outline their further benefits.

Theoretical Economics,Cryptography and Security
An Empirical Study on Real Bug Fixes

Hao Zhong,Zhendong Su

DOI: https://doi.org/10.1109/icse.2015.101

2015-01-01

Abstract:Software bugs can cause significant financial loss and even the loss of human lives. To reduce such loss, developers devote substantial efforts to fixing bugs, which generally requires much expertise and experience. Various approaches have been proposed to aid debugging. An interesting recent research direction is automatic program repair , which achieves promising results, and attracts much academic and industrial attention. However, people also cast doubt on the effectiveness and promise of this direction. A key criticism is to what extent such approaches can fix real bugs. As only research prototypes for these approaches are available, it is infeasible to address the criticism by evaluating them directly on real bugs. Instead, in this paper, we design and develop BugStat, a tool that extracts and analyzes bug fixes. With BugStat's support, we conduct an empirical study on more than 9,000 real-world bug fixes from six popular Java projects. Comparing the nature of manual fixes with automatic program repair, we distill 15 findings, which are further summarized into four insights on the two key ingredients of automatic program repair: fault localization and faulty code fix. In addition, we provide indirect evidence on the size of the search space to fix real bugs and find that bugs may also reside in non-source files. Our results provide useful guidance and insights for improving the state-of-the-art of automatic program repair.
MyCites: a proposal to mark and report inaccurate citations in scholarly publications

Mohammad Hosseini,Martin Paul Eve,Bert Gordijn,Cameron Neylon

DOI: https://doi.org/10.1186/s41073-020-00099-8

2020-09-17

Research Integrity and Peer Review

Abstract:Abstract Background Inaccurate citations are erroneous quotations or instances of paraphrasing of previously published material that mislead readers about the claims of the cited source. They are often unaddressed due to underreporting, the inability of peer reviewers and editors to detect them, and editors’ reluctance to publish corrections about them. In this paper, we propose a new tool that could be used to tackle their circulation. Methods We provide a review of available data about inaccurate citations and analytically explore current ways of reporting and dealing with these inaccuracies. Consequently, we make a distinction between publication (i.e., first occurrence) and circulation (i.e., reuse) of inaccurate citations. Sloppy reading of published items, literature ambiguity and insufficient quality control in the editorial process are identified as factors that contribute to the publication of inaccurate citations. However, reiteration or copy-pasting without checking the validity of citations, paralleled with lack of resources/motivation to report/correct inaccurate citations contribute to their circulation. Results and discussion We propose the development of an online annotation tool called “MyCites” as means with which to mark and map inaccurate citations. This tool allows ORCID users to annotate citations and alert authors (of the cited and citing articles) and also editors of journals where inaccurate citations are published. Each marked citation would travel with the digital version of the document (persistent identifiers) and be visible on websites that host peer-reviewed articles (journals’ websites, Pubmed, etc.). In the future development of MyCites, challenges such as the conditions of correct/incorrect-ness and parties that should adjudicate that, and, the issue of dealing with incorrect reports need to be addressed.
Sixty seconds on . . . spelling errors

Matthew Limb

DOI: https://doi.org/10.1136/bmj.q2414

2024-11-02

BMJ

Abstract:Doctor, I've been reading about pubic health and I'm worriedYou mean public health?What happened there?You've found the error strewn world of academia.But publishing standards are rising, surely?Not according to analysis of over 32 million research abstracts published over the past 50 years. Eleven of 15 common spelling errors have increased over time, with most showing a strong increase. The total error rate has jumped from 0.1 per 10 000 abstracts in 1970 to 8.7 per 10 000 in 2023.How do we know?Statistics experts Adrian Barnett and Nicole White, from Queensland University of Technology, Australia, analysed trends over time.1 They'd already found researchers were misusing statistical methods then spotted some couldn't even get the names of the methods right—such as "Fischer's exact test" instead of "Fisher's exact test."How bad is it?Clearly less serious than other bad research writing practices such as plagiarism and spin—however, "We think it's bad not because we...

medicine, general & internal
Is biomedical research self-correcting? Modelling insights on the persistence of spurious science

David Robert Grimes

DOI: https://doi.org/10.1098/rsos.231056

IF: 3.5

2024-01-01

Royal Society Open Science

Abstract:The reality that volumes of published biomedical research are not reproducible is an increasingly recognized problem. Spurious results reduce trustworthiness of reported science, increasing research waste. While science should be self-correcting from a philosophical perspective, that in insolation yields no information on efforts required to nullify suspect findings or factors shaping how quickly science may be corrected. There is also a paucity of information on how perverse incentives in the publishing ecosystem favouring novel positive findings over null results shape the ability of published science to self-correct. Knowledge of factors shaping self-correction of science remain obscure, limiting our ability to mitigate harms. This modelling study introduces a simple model to capture dynamics of the publication ecosystem, exploring factors influencing research waste, trustworthiness, corrective effort and time to correction. Results from this work indicate that research waste and corrective effort are highly dependent on field-specific false positive rates and time delays to corrective results to spurious findings are propagated. The model also suggests conditions under which biomedical science is self-correcting and those under which publication of correctives alone cannot stem propagation of untrustworthy results. Finally, this work models a variety of potential mitigation strategies, including researcher- and publisher-driven interventions.

multidisciplinary sciences
Reproducibility: A tragedy of errors

David B. Allison,Andrew W. Brown,Brandon J. George,Kathryn A. Kaiser

DOI: https://doi.org/10.1038/530027a

IF: 64.8

2016-02-03

Nature

Abstract:Mistakes in peer-reviewed papers are easy to find but hard to fix, report David B. Allison and colleagues.

multidisciplinary sciences
Cut to the Chase: an Error-Oriented Approach to Detect Error-Handling Bugs

Haoran Liu,Zhouyang Jia,Shanshan Li,Yan Lei,Yue Yu,Yu Jiang,Xiaoguang Mao,Xiangke Liao

DOI: https://doi.org/10.1145/3660787

2024-01-01

Abstract:Error-handling bugs are prevalent in software systems and can result in severe consequences. Existing works on error-handling bug detection can be categorized into template-based and learning-based approaches. The former requires much human effort and is difficult to accommodate the software evolution. The latter usually focuses on errors of API and assumes that error handling should be right after the handled error. Such an assumption, however, may affect both learning and detecting phases. The existing learning-based approaches can be regarded as API-oriented, which starts from an API and learns if the API requires error handling. In this paper, we propose EH-Digger, an ERROR-oriented approach, which starts from an error handling. Our approach can learn why the error occurs and when the error has to be handled. We conduct a comprehensive study on 2,322 error-handling code snippets from 22 widely used software systems across 8 software domains to reveal the limitation of existing approaches and guide the design of EH-Digger. We evaluated EH-Digger on the Linux Kernel and 11 open-source applications. It detected 53 new bugs confirmed by the developers and 71 historical bugs fixed in the latest versions. We also compared EH-Digger with three state-of-the-art approaches, 30.1% of bugs detected by EH-Digger cannot be detected by the existing approaches.
Detecting Error-Handling Bugs without Error Specification Input.

Zhouyang Jia,Shanshan Li,Tingting Yu,Xiangke Liao,Ji Wang,Xiaodong Liu,Yunhuai Liu

DOI: https://doi.org/10.1109/ASE.2019.00029

2019-01-01

Abstract:Most software systems frequently encounter errors when interacting with their environments. When errors occur, error-handling code must execute flawlessly to facilitate system recovery. Implementing correct error handling is repetitive but non-trivial, and developers often inadvertently introduce bugs into error-handling code. Existing tools require correct error specifications to detect error-handling bugs. Manually generating error specifications is error-prone and tedious, while automatically mining error specifications is hard to achieve a satisfying accuracy. In this paper, we propose EH-Miner, a novel and practical tool that can automatically detect error-handling bugs without the need for error specifications. Given a function, EH-Miner mines its error-handling rules when the function is frequently checked by an equivalent condition, and handled by the same action. We applied EH-Miner to 117 applications across 15 software domains. EH-Miner mined error-handling rules with the precision of 91.1% and the recall of 46.9%. We reported 142 bugs to developers, and 106 bugs had been confirmed and fixed at the time of writing. We further applied EH-Miner to Linux kernel, and reported 68 bugs for kernel-4.17, of which 42 had been confirmed or fixed.
Dear researchers: send us your lab hacks

Jack Leeming

DOI: https://doi.org/10.1038/d41586-024-03217-4

IF: 64.8

2024-10-10

Nature

Abstract:Do you have a favourite tool or workaround that helps you save time, money or drudgery? Share it with Nature's readers. Do you have a favourite tool or workaround that helps you save time, money or drudgery? Share it with Nature's readers.

multidisciplinary sciences
CRSearcher: Searching Code Database for Repairing Bugs.

Yingyi Wang,Yuting Chen,Beijun Shen,Hao Zhong

DOI: https://doi.org/10.1145/3131704.3131720

2017-01-01

Abstract:With the exponentially rising of software development in the past decades, millions of software products have been created. Existing empirical studies show that many code snippets are similar. Although there exist many difficulties in maintaining these similar code snippets, we believe that it is feasible to leverage the similarity to enhance program repairs, as bugs may have already been repaired in many other similar code snippets. In this paper, we propose CRSearcher, an approach, that searches open-sourced codebases and uses similar code to repair bugs. It is designed with insights from three observations: small patch, code redundancy, and the availability of open-sourced software repositories. With these insights, CRSearcher provides the engineers with an interactive strategy for program repair which automatically locates bug location, searches for similar code, and recommends code snippets as an aid, but allows developers to make a final decision. We have implemented CRSearcher on the basis of the state-of-the-art techniques such as Findbugs and a token-based similar code snippets search engine. In our evaluation, we have built a code database that consists of several open-source projects. Our evaluation results show that CRSearcher does help generating high quality program patches, and reduce more than 50% time on repairing six buggy programs.
CODECHECK: An open-science initiative to facilitate sharing of computer programs and results presented in scientific publications

Stephen Eglen,Daniel Nüst

DOI: https://doi.org/10.7557/5.4910

2019-09-20

Septentrio Conference Series

Abstract:Analysis of data and computational modelling is central to most scientific disciplines. The underlying computer programs are complex and costly to design. However, these computational techniques are rarely checked during review of the corresponding papers, nor shared upon publication. Instead, the primary method for sharing data and computer programs today is for authors to state "data available upon reasonable request", although the actual code and data is the only sufficiently detailed description of a computational workflow that allows reproduction and reuse. Despite best intentions, these programs and data can quickly disappear from laboratories. Furthermore, there is a reluctance to share: only 8% of papers in recent top-tier AI conferences shared code relating to their publications (Gundersen et al. 2018). This low-rate of code sharing is seen in other fields, e.g. computational physics (Stodden et al. 2018). Given that code and data are rich digital artefacts that can be shared relatively easily, and that funders and journal publishers increasingly mandate sharing of resources, we should be sharing more and follow best practices for data and software publication. The permanent archival of valuable code and datasets would allow other researchers to make use of these resources in their work, and improve the reliability of reporting as well as the quality of tools. We are building a computational platform, called CODECHECK (http://www.codecheck.org.uk), to enhance the availability, discovery and reproducibility of published computational research. Researchers that provide code and data will have their code independently run to ensure the computational parts of a workflow can be reproduced. The results from our independent run will then be shared freely post-publication in an open repository. The reproduction is attributed to the person perfoming the check. Our independent runs will act as a "certificate of reproducible computation". These certificates will be of use to several parties at different times during the generation of a scientific publication. Prior to peer review, the researchers themselves can check that their code runs on a separate platform. During peer review, editors and reviewers can check if the figures in the certificate match those presented in manuscripts for review without cumbersome download and installation procedures. Once published, any interested reader can download the software and even data that was used to generate the results shown in the certificate. The code and results from papers are shared according to the principles we recently outlined (Eglen et al. 2017). To ensure our system scales to large numbers of papers and is trustworthy, our system will be as automated as possible, fully open itself, and rely on open source software and open scholarly infrastructure. This presentation will discuss the challenges faced to date in building the system and in connecting it with existing peer-review principles, and plans for links with open access journals. Acknolwedgements This work has been funded by the UK Software Sustainability Institute, a Mozilla Open Science Mini grant and the German Research Foundation (DFG) under project number PE 1632/17-1.
Detecting Bugs by Discovering Expectations and Their Violations

Pan Bian,Bin Liang,Yan Zhang,Chaoqun Yang,Wenchang Shi,Yan Cai

DOI: https://doi.org/10.1109/tse.2018.2816639

IF: 7.4

2019-01-01

IEEE Transactions on Software Engineering

Abstract:Code mining has been proven to be a promising approach to inferring implicit programming rules for finding software bugs. However, existing methods may report large numbers of false positives and false negatives. In this paper, we propose a novel approach called EAntMiner to improve the effectiveness of code mining. EAntMiner elaborately reduces noises from statements irrelevant to interesting rules and different implementation forms of the same logic. During preprocessing, we employ program slicing to decompose the original source repository into independent sub-repositories. In each sub-repository, statements irrelevant to critical operations (automatically extracted from source code) are excluded and various semantics-equivalent implementations are normalized into a canonical form as far as possible. Moreover, to tackle the challenge that some bugs are difficult to be detected by mining frequent patterns as rules, we further developed a kNN-based method to identify them. We have implemented EAntMiner and evaluated it on four large-scale C systems. EAntMiner successfully detected 105 previously unknown bugs that have been confirmed by corresponding development communities. A set of comparative evaluations also demonstrate that EAntMiner can effectively improve the precision of code mining.
Given enough eyeballs, all bugs are shallow? Revisiting Eric Raymond with bug bounty programs

Thomas Maillart,Mingyi Zhao,Jens Grossklags,John Chuang

DOI: https://doi.org/10.1093/cybsec/tyx008

2017-06-01

Journal of Cyber Security

Abstract:Bug bounty programs offer a modern way for organizations to crowdsource their software security, and for security researchers to be fairly rewarded for the vulnerabilities they find. However, little is known on the incentives set by bug bounty programs – how they drive engagement and new bug discoveries. This article provides an empirical investigation of the strategic interactions among the managers and participants of bug bounty programs, as well as the intermediation by bug bounty platforms. We find that for a given bug bounty program, each security researcher can only expect to discover a bounded number of bugs. This result offers a validation step to a theory brought forth early on by Brady et al. This theory proposes that each security researcher inspecting a piece of software offers a unique environment of skills and mindset, which is amenable to the discovery of bugs that others may not be able to uncover. Bug bounty programs indeed benefit from the engagement of large crowds of researchers. Conversely, security researchers benefit greatly from searching for bugs in multiple bug bounty programs. However, we find that following a strong front-loading effect, newly launched programs attract researchers at the expense of older programs: the probability of finding bugs decays as ∼1/t0.4 after the launch of a program, even though bugs found later yield on average higher rewards. Our results lead us to formulate three recommendations for organizing bug bounty programs and platforms: (i) organize enrollment, mobility, and renewal of security researchers across bounty programs, (ii) highlight and organize programs for front-loading, and (iii) organize fluid market transactions to reduce uncertainty and thus reduce incentives for security researchers to sell on the black market.

English Else
Understanding 'error' in the forensic sciences: A primer

Kristy A Martire,Jason M Chin,Carolyn Davis,Gary Edmond,Bethany Growns,Stacey Gorski,Richard I Kemp,Zara Lee,Christopher M Verdon,Gabrielle Jansen,Tanya Lang,Tess M S Neal,Rachel A Searston,Joshua Slocum,Stephanie Summersby,Jason M Tangen,Matthew B Thompson,Alice Towler,Darren Watson,Melissa V Werrett,Mariam Younan,Kaye N Ballantyne

DOI: https://doi.org/10.1016/j.fsisyn.2024.100470

2024-04-29

Abstract:This paper distils seven key lessons about 'error' from a collaborative webinar series between practitioners at Victoria Police Forensic Services Department and academics. It aims to provide the common understanding of error necessary to foster interdisciplinary dialogue, collaboration and research. The lessons underscore the inevitability, complexity and subjectivity of error, as well as opportunities for learning and growth. Ultimately, we argue that error can be a potent tool for continuous improvement and accountability, enhancing the reliability of forensic sciences and public trust. It is hoped the shared understanding provided by this paper will support future initiatives and funding for collaborative developments in this vital domain.
Decontamination of the scientific literature

Guillaume Cabanac

DOI: https://doi.org/10.48550/arXiv.2210.15912

2022-10-28

Digital Libraries

Abstract:Research misconduct and frauds pollute the scientific literature. Honest errors and malevolent data fabrication, image manipulation, journal hijacking, and plagiarism passed peer review unnoticed. Problematic papers deceive readers, authors citing them, and AI-powered literature-based discovery. Flagship publishers accepted hundreds flawed papers despite claiming to enforce peer review. This application ambitions to decontaminate the scientific literature using curative and preventive actions.
Illuminating 'the ugly side of science': fresh incentives for reporting negative results

Rachel Brazil

DOI: https://doi.org/10.1038/d41586-024-01389-7

IF: 64.8

2024-05-09

Nature

Abstract:New data repositories and alternative journals and workshops offer routes for sharing negative results — which could help to solve the reproducibility crisis and give machine learning a boost. New data repositories and alternative journals and workshops offer routes for sharing negative results — which could help to solve the reproducibility crisis and give machine learning a boost.

multidisciplinary sciences
Learning from error

V. Gregson

DOI: https://doi.org/10.5840/method19831225

2023-07-25

Development and Learning in Organizations: An International Journal

Abstract:Purpose This paper aims to review the latest management developments across the globe and pinpoint practical implications from cutting-edge research and case studies. Design/methodology/approach This briefing is prepared by an independent writer who adds their own impartial comments and places the articles in context. Findings Errors are becoming a source of valuable learning opportunities. Companies can increase their ability to learn from error by focusing on the development of the key psychological attributes learning goal orientation, motivation to learn and metacognition. Originality/value The briefing saves busy executives and researchers hours of reading time by selecting only the very best, most pertinent information and presenting it in a condensed and easy-to-digest format.

Biology,Business

Cash for catching scientific errors

Pay researchers to spot errors in published papers

Facing the Facts

An Empirical Study of End-User Behaviour in Spreadsheet Error Detection & Correction

Artificial Bugs for Crowdsearch

An Empirical Study on Real Bug Fixes

MyCites: a proposal to mark and report inaccurate citations in scholarly publications

Sixty seconds on . . . spelling errors

Is biomedical research self-correcting? Modelling insights on the persistence of spurious science

Reproducibility: A tragedy of errors

Cut to the Chase: an Error-Oriented Approach to Detect Error-Handling Bugs

Detecting Error-Handling Bugs without Error Specification Input.

Dear researchers: send us your lab hacks

CRSearcher: Searching Code Database for Repairing Bugs.

CODECHECK: An open-science initiative to facilitate sharing of computer programs and results presented in scientific publications

Detecting Bugs by Discovering Expectations and Their Violations

Given enough eyeballs, all bugs are shallow? Revisiting Eric Raymond with bug bounty programs

Understanding 'error' in the forensic sciences: A primer

Decontamination of the scientific literature

Illuminating 'the ugly side of science': fresh incentives for reporting negative results

Learning from error