The importance of documenting failure

Pietro Barabaschi

DOI: https://doi.org/10.1038/s42254-024-00702-7

IF: 36.273

2024-02-27

Nature Reviews Physics

Abstract:Pietro Barabaschi, Director General of ITER, calls for measures and incentives to carefully document the entire research process, including dead ends and failures, instead of reporting just the successful final results.

physics, applied, multidisciplinary

What problem does this paper attempt to address?

Creativity is Allowing for Failures: Leader Failure Tolerance and Employee Creativity

Shenjiang Mo,Siu Yin Cheung,Yaping Gong

DOI: https://doi.org/10.5465/ambpp.2019.17258abstract

2019-01-01

Abstract:The creative process is inherently risky and prone to failure. To promote creativity, scholars and practitioners have advocated the importance of having team leaders who are tolerant of employees’ failures in creative activities. However, creativity research thus far has not explicitly examined the role of leader failure tolerance. To address this knowledge void, from the systems view of creativity we examine a cognitive mechanism known as bisociation, through which leader failure tolerance may influence the production of novel and useful ideas. We also examine the joint effects of leader failure tolerance and individual differences in goal orientation on creativity through bisociation. To achieve our research goal, we first develop and validate a measure of leader failure tolerance using two independent samples. Then, we test the hypothesized relationships using a sample of 210 research and development (R&D) teams comprising 632 employees. Results show that leader failure tolerance positively relates to employee creativity through bisociation. Furthermore, when leader failure tolerance is high, performance-proving goal orientation has a positive indirect effect on creativity via bisociation, while performance-avoiding goal orientation has a negative indirect effect on creativity via bisociation. Taken together, these insights have important implications for managers charged with delivering creativity.
Success and Failure in Software Engineering: a Followup Systematic Literature Review

Damian A. Tamburri,Fabio Palomba,Rick Kazman

DOI: https://doi.org/10.48550/arXiv.2006.12086

2020-06-22

Abstract:Success and failure in software engineering are still among the least understood phenomena in the discipline. In a recent special journal issue on the topic, Mantyla et al. started discussing these topics from different angles; the authors focused their contributions on offering a general overview of both topics without deeper detail. Recognising the importance and impact of the topic, we have executed a followup, more in-depth systematic literature review with additional analyses beyond what was previously provided. These new analyses offer: (a) a grounded-theory of success and failure factors, harvesting over 500+ factors from the literature; (b) 14 manually-validated clusters of factors that provide relevant areas for success- and failure-specific measurement and risk-analysis; (c) a quality model composed of previously unmeasured organizational structure quantities which are germane to software product, process, and community quality. We show that the topics of success and failure deserve further study as well as further automated tool support, e.g., monitoring tools and metrics able to track the factors and patterns emerging from our study. This paper provides managers with risks as well as a more fine-grained analysis of the parameters that can be appraised to anticipate the risks.

Software Engineering
Illuminating 'the ugly side of science': fresh incentives for reporting negative results

Rachel Brazil

DOI: https://doi.org/10.1038/d41586-024-01389-7

IF: 64.8

2024-05-09

Nature

Abstract:New data repositories and alternative journals and workshops offer routes for sharing negative results — which could help to solve the reproducibility crisis and give machine learning a boost. New data repositories and alternative journals and workshops offer routes for sharing negative results — which could help to solve the reproducibility crisis and give machine learning a boost.

multidisciplinary sciences
Mid-career pitfall of consecutive success in science

Noriyuki Higashide,Takahiro Miura,Yuta Tomokiyo,Kimitaka Asatani,Ichiro Sakata

2024-03-09

Abstract:The creativity of scientists often manifests as localized hot streaks of significant success. Understanding the underlying mechanisms of these influential phases can enhance the effectiveness of support systems and funding allocation, fostering groundbreaking discoveries worthy of accolades. Historically, analyses have suggested that hot streaks occur randomly over time. However, our research, through meticulous examination, reveals that these phases are not flatly distributed but are more frequent at the early and late stages of scientists' careers. Notably, both early and late hot streaks are marked by dense tie collaborations, with the former typically involving close partnerships with particular authors and the latter being characterized by involvement in large-scale projects compared with single-top or ordinary papers. This pattern indicates that mid-career researchers lack both intimate relations and resources to keep big projects, leading to``mid-career pitfal'' of consecutive success. This insight holds profound implications for the development of policies and initiatives aimed at bolstering innovative research and discovery.

Social and Information Networks,Physics and Society
Reflections on Software Failure Analysis

Paschal C. Amusuo,Aishwarya Sharma,Siddharth R. Rao,Abbey Vincent,James C. Davis

DOI: https://doi.org/10.1145/3540250.3560879

2022-09-22

Abstract:Failure studies are important in revealing the root causes, behaviors, and life cycle of defects in software systems. These studies either focus on understanding the characteristics of defects in specific classes of systems or the characteristics of a specific type of defect in the systems it manifests in. Failure studies have influenced various software engineering research directions, especially in the area of software evolution, defect detection, and program repair. In this paper, we reflect on the conduct of failure studies in software engineering. We reviewed a sample of 52 failure study papers. We identified several recurring problems in these studies, some of which hinder the ability of the engineering community to trust or replicate the results. Based on our findings, we suggest future research directions, including identifying and analyzing failure causal chains, standardizing the conduct of failure studies, and tool support for faster defect analysis.

Software Engineering
Designing for Reproducibility: A Qualitative Study of Challenges and Opportunities in High Energy Physics

Sebastian S. Feger,Sünje Dallmeier-Tiessen,Albrecht Schmidt,Paweł W. Woźniak

DOI: https://doi.org/10.1145/3290605.3300685

2019-03-14

Abstract:Reproducibility should be a cornerstone of scientific research and is a growing concern among the scientific community and the public. Understanding how to design services and tools that support documentation, preservation and sharing is required to maximize the positive impact of scientific research. We conducted a study of user attitudes towards systems that support data preservation in High Energy Physics, one of science's most data-intensive branches. We report on our interview study with 12 experimental physicists, studying requirements and opportunities in designing for research preservation and reproducibility. Our findings suggest that we need to design for motivation and benefits in order to stimulate contributions and to address the observed scalability challenge. Therefore, researchers' attitudes towards communication, uncertainty, collaboration and automation need to be reflected in design. Based on our findings, we present a systematic view of user needs and constraints that define the design space of systems supporting reproducible practices.

Human-Computer Interaction,High Energy Physics - Experiment
The impact of failure on faculty development

Rahul Nohria,Sarah McBane

DOI: https://doi.org/10.1016/j.cptl.2021.11.016

Abstract:Introduction: Failure is an inevitable outcome in life. Even though this is a commonly accepted fact, the environment in academic pharmacy has stigmatized failure. This stigmatization potentially stunts faculty development if inappropriate coping strategies evolve and skills such as creativity and problem solving do not flourish. Commentary: There is an opportunity for pharmacy academia, especially faculty, to assume a new lens when examining failure. The authors will examine the impact of failure on active learning and scholarship. Implication: Despite the many challenges and apprehensions that surround the concept of failure, it is important that academia reframes failure. This new perspective can positively view failure as a progressive mechanism by creating novel solutions to challenges in academia and demonstrating to pharmacy students that failure is something to embrace as a learning tool.
Deceiving scientific research, misconduct events are possibly a more common practice than foreseen

Alonzo Alfaro-Núñez

DOI: https://doi.org/10.1186/s12302-022-00659-3

Abstract:Background: Today, scientists and academic researchers experience an enormous pressure to publish innovative and ground-breaking results in prestigious journals. This pressure may blight the general view concept of how scientific research needs to be done in terms of the general rules of transparency; duplication of data, and co-authorship rights might be compromised. As such, misconduct acts may occur more frequently than foreseen, as frequently these experiences are not openly shared or discussed among researchers. Main body: While there are some concerns about the health and the transparency implications of such normalised pressure practices imposed on researchers in scientific research, there is a general acceptance that researchers must take and accept it in order to survive in the competitive world of science. This is even more the case for junior and mid-senior researchers who have recently started their adventure into the universe of independent researchers. Only the slightest fraction manages to endure, after many years of furious and cruel rivalry, to obtain a long-term, and even less probable, permanent position. There is an evil circle; excellent records of good publications are needed in order to obtain research funding, but how to produce pioneering research during these first years without funding? Many may argue this is a necessary process to ensure good quality scientific investigation, possibly, but perseverance and resilience may not be the only values needed when rejection is received consecutively for years. Conclusion: There is a general culture that scientists rarely share previous bad experiences, in particular if they were associated to misconduct, as they may not be seen or considered as a relevant or hot topic to the scientific community readers. On next, a recent misconduct experience is shared, and a few additional reflections and suggestions on this topic were drafted in the hope other researchers might be spared unnecessary and unpleasant times.
Quantifying Success in Science: An Overview

Xiaomei Bai,Hanxiao Pan,Jie Hou,Teng Guo,Ivan Lee,Feng Xia

DOI: https://doi.org/10.48550/arXiv.2008.04649

2020-08-10

Abstract:Quantifying success in science plays a key role in guiding funding allocations, recruitment decisions, and rewards. Recently, a significant amount of progresses have been made towards quantifying success in science. This lack of detailed analysis and summary continues a practical issue. The literature reports the factors influencing scholarly impact and evaluation methods and indices aimed at overcoming this crucial weakness. We focus on categorizing and reviewing the current development on evaluation indices of scholarly impact, including paper impact, scholar impact, and journal impact. Besides, we summarize the issues of existing evaluation methods and indices, investigate the open issues and challenges, and provide possible solutions, including the pattern of collaboration impact, unified evaluation standards, implicit success factor mining, dynamic academic network embedding, and scholarly impact inflation. This paper should help the researchers obtaining a broader understanding of quantifying success in science, and identifying some potential research directions.

Digital Libraries,Physics and Society
When failure fails to be productive: probing the effectiveness of productive failure for learning beyond STEM domains

Valentina Nachtigall,Katja Serova,Nikol Rummel

DOI: https://doi.org/10.1007/s11251-020-09525-2

2020-11-19

Instructional Science

Abstract:Abstract The current work builds on research demonstrating the effectiveness of Productive Failure (PF) for learning. While the effectiveness of PF has been demonstrated for STEM learning, it has not yet been investigated whether PF is also beneficial for learning in non-STEM domains. Given this need to test PF for learning in domains other than mathematics or science, and the assumption that features embodied in a PF design are domain-independent, we investigated the effect of PF on learning social science research methods. We conducted two quasi-experimental studies with 212 and 152 10th graders. Following the paradigm of typical PF studies, we implemented two conditions: PF, in which students try to solve a complex problem prior to instruction, and Direct Instruction (DI), in which students first receive instruction followed by problem solving. In PF, students usually learn from their failure. Failing to solve a complex problem is assumed to prepare students for deeper learning from subsequent instruction. In DI, students usually learn through practice. Practicing and applying a given problem-solving procedure is assumed to help students to learn from previous instruction. In contrast to several studies demonstrating beneficial effects of PF on learning mathematics and science, in the present two studies, PF students did not outperform DI students on learning social science research methods. Thus, the findings did not replicate the PF effect on learning in a non-STEM domain. The results are discussed in light of mechanisms assumed to underlie the benefits of PF.

education & educational research,psychology, educational
Teaching about the birth of synchrotron light: the role of Frascati and a missed opportunity

G. Margaritondo

DOI: https://doi.org/10.1107/s160057752400300x

IF: 2.557

2024-05-23

Journal of Synchrotron Radiation

Abstract:A collection of personal reminiscences illuminates many widely unknown facts about the first steps of synchrotron radiation research; in particular, the events that brought Frascati close to world leadership in this domain.The users of synchrotron light are now tens of thousands throughout the world. Paradoxically, many of them do not know much about the early history of their domain. This is regrettable, since education about the initial developments makes it easier to fully understand synchrotron radiation and effectively use its amazing features. Scarcely known, in particular, is the key role of scientists working in Frascati, Italy. Partly based on his personal experiences, the author reports here relevant aspects of this story, including a pioneering French–Italian experiment that started in the early 1960s, and the Frascati contributions in the 1970s and 1980s to the birth of synchrotron light research. Finally, the unwise strategic decisions that prevented Italy from achieving absolute leadership in this domain – in spite of its unique initial advantages – are analyzed.

optics,physics, applied,instruments & instrumentation
Documenting research software in engineering science

Sibylle Hermann,Jörg Fehr

DOI: https://doi.org/10.1038/s41598-022-10376-9

IF: 4.6

2022-04-21

Scientific Reports

Abstract:Abstract The reuse of research software needs good documentation, however, the documentation in particular is often criticized. Especially in non-IT specific disciplines, the lack of documentation is attributed to the lack of training, the lack of time or missing rewards. This article addresses the hypothesis that scientists do document but do not know exactly what they need to document, why, and for whom. In order to evaluate the actual documentation practice of research software, we examined existing recommendations, and we evaluated their implementation in everyday practice using a concrete example from the engineering sciences and compared the findings with best practice examples. To get a broad overview of what documentation of research software entailed, we defined categories and used them to conduct the research. Our results show that the big picture of what documentation of research software means is missing. Recommendations do not consider the important role of researchers, who write research software, whose documentation takes mainly place in their research articles. Moreover, we show that research software always has a history that influences the documentation.

multidisciplinary sciences
Negative equity - the value of reporting negative results

Owen Sansom,Debora Bogani,Linus Reichenbach,Sara Wells

DOI: https://doi.org/10.1242/dmm.050937

2024-08-01

Abstract:A pervasive discussion point within the scientific community is the value of unpublished or unavailable data. Researchers, funders, ethical review bodies, editors and publishers have all highlighted the need to make more data available to enhance experimental planning and interpretation and to prevent others from repeating similar experiments. This is particularly important in the context of experimentation involving animals and efforts towards replacement, refinement and reduction. However, despite this broad agreement, sharing data that show inconclusive, statistically insignificant or unremarkable results is still not common practice. In this Editorial, we will highlight the value of what are often coined negative (or null) data and outline some emerging initiatives to address the gap between data generated in laboratories and data available to the wider scientific community.
FAIL Is Not a Four-Letter Word: A Theoretical Framework for Exploring Undergraduate Students’ Approaches to Academic Challenge and Responses to Failure in STEM Learning Environments

Meredith A. Henry,Shayla Shorter,Louise Charkoudian,Jennifer M. Heemstra,Lisa A. Corwin

DOI: https://doi.org/10.1187/cbe.18-06-0108

2019-03-01

CBE—Life Sciences Education

Abstract:Navigating scientific challenges, persevering through difficulties, and coping with failure are considered hallmarks of a successful scientist. However, relatively few studies investigate how undergraduate science, technology, engineering, and mathematics (STEM) students develop these skills and dispositions or how instructors can facilitate this development in undergraduate STEM learning contexts. This is a critical gap, because the unique cultures and practices found in STEM classrooms are likely to influence how students approach challenges and deal with failures, both during their STEM education and in the years that follow. To guide research aimed at understanding how STEM students develop a challenge-engaging disposition and the ability to adaptively cope with failure, we generate a model representing hypotheses of how students might approach challenges and respond to failures in undergraduate STEM learning contexts. We draw from theory and studies investigating mindset, goal orientations, attributions, fear of failure, and coping to inform our model. We offer this model as a tool for the community to test, revise, elaborate, or refute. Finally, we urge researchers and educators to consider the development, implementation, and rigorous testing of interventions aimed at helping students develop a persevering and challenge-engaging disposition within STEM contexts.
Promoting and Supporting Innovation Through Collaborative Research

Jim Gentile

DOI: https://doi.org/10.1096/fasebj.26.1_supplement.617.1

2012-04-01

The FASEB Journal

Abstract:Many of the major scientific challenges of the 21st century are likely to require collaborative and interdisciplinary teams. Creating good “team players” who can work synergistically to undertake high‐risk, high‐reward research should be a priority, and demonstrating success in collaborations should be encouraged by private and federal granting agencies, and more highly rewarded by universities in tenure and promotion decisions. Given the challenges that our nation faces, tenure and promotion decisions should no longer be based merely on the number of papers published or overhead dollars generated. Journal editorial boards and foundation peer‐review panels are essential to the highest‐quality science, but they should not be viewed as surrogate members of tenure and promotion committees. Rewarding careers paths should be clearly visible to the nation's newly minted science Ph.D.s, postdoctoral researchers and “tenure track” professors. Allowing these early‐career scientists more expansive opportunities during their most creative years would make the United States, and the academic communities in which they serve, an increasingly more vibrant and creative environments. The heart and soul of American science and enterprise is the willingness to take risks, to learn from failure and to repeat the process until success is achieved. Our early‐career scientists should have the same opportunity.

cell biology,biochemistry & molecular biology,biology
Reflections on Failure Education to Foster Creativity in the Age of Innovation

Hwasun Lee,

DOI: https://doi.org/10.36358/jce.2023.23.3.21

2023-09-30

Korean Society for Creativity Education

Abstract:Failure is an integral part of creativity and an essential part of preparing for the adaptation and flexibility needed in an environment of accelerating technological change. However, the link between creativity and failure in creativity research and education is still underrepresented or missing compared to its importance. This study aims to orient failure education to foster creativity by starting with the question: can failure play a positive role in teaching and learning creativity, and if so, what pedagogical methods can be applied? This study is organized into three phases. First, the significance and role of failure in the era of creativity-based innovation were examined theoretically. Second, the concept of failure and innovative methodologies related to creativity were summarized, and finally, five directions of failure education were proposed by extracting key factors based on previous studies. The understanding of creative failure education will provide a foundation for the discussion and revitalization of failure education in the era of innovation.
Why (and how) we should publish negative data

Simon Nimpf,David A Keays

DOI: https://doi.org/10.15252/embr.201949775

IF: 9.071

2019-12-20

EMBO Reports

Abstract:Central to the scientific method are the concepts of falsifiability and hypothesis testing: We cannot prove a hypothesis, merely acquire evidence that supports or refutes it <a class="bibLink tab-link" href="#embr201949775-bib-0001">1</a>. We should reject those ideas that we can refute and advance those that we cannot. In this respect, the generation and publication of negative data lie at the very heart of the scientific enterprise, and yet, there is an overwhelming focus on positive findings. Whether it is a PhD student, postdoc, lab head or an editor, there is a great reluctance to publish a paper that begins with the word "No". In fact, the percentage of papers declaring support for a tested hypothesis has increased by 22% between 1990 and 2007 across scientific disciplines <a class="bibLink tab-link" href="#embr201949775-bib-0002">2</a>. This has served to fuel the exaggeration and distortion of scientific findings, which has led to the "reproducibility crisis". Current evidence suggests that between 51 and 89% of published studies cannot be reproduced <a class="bibLink tab-link" href="#embr201949775-bib-0003">3</a><a class="bibLink tab-link" href="#embr201949775-bib-0004">4</a><a class="bibLink tab-link" href="#embr201949775-bib-0005">5</a>, a fact which has even caught the attention of the mainstream media. It has considerable economic consequences, resulting in some $28 billion of wasteful spending per year in the USA alone <a class="bibLink tab-link" href="#embr201949775-bib-0006">6</a>. Failure to publish negative data ensures that dubious ideas and wrong‐headed projects receive financial support with multiple groups toiling away in vain, when that money and time could be spent on more productive endeavors <a class="bibLink tab-link" href="#embr201949775-bib-0007">7</a>. So, the philosophical, practical, and economic arguments for publishing negative data are strong—but what is the best way to do this?

cell biology,biochemistry & molecular biology
The Role of Failure in the Entrepreneurial Process: A Systematic Literature Review

Alessia Pisoni,Emanuele Aversa Aversa,Alberto Onetti

DOI: https://doi.org/10.5539/ijbm.v16n1p53

2020-12-14

International Journal of Business and Management

Abstract:Failure is a crucial event that can occur at any time during the entrepreneurial/start-up process. Understanding what influences the failure or survival of new ventures is increasingly attracting the interest of scholars, practitioners, and policymakers, mainly because of the role that startups play in innovation. Studying failure events presents a series of challenges that scholars should bear in mind when approaching this topic, starting from the definition of terms to the lack of data to analyze such events. The literature on business failures is scattered among different fields of research and lacks a comprehensive framework. We address this gap performing a systematic literature review. 74 papers focusing on new ventures’ failure have been reviewed and analyzed to identify the main causes of failure. In doing so, we identify four main categories of causes of new venture failure. Namely, I) resources, with a specific focus on human and financial capital; II) strategic/managerial decisions; III) product-related aspects; and IV) contextual/environmental-related issues. By providing an up-to-date systematization of recently published contributions on the topic, we aim to provide practical implications for entrepreneurs/practitioners and future research directions to researchers in the field.
Documenting research in simulation science to enhance understanding for reusability

Sibylle Hermann,Jörg Fehr

DOI: https://doi.org/10.1098/rsos.240776

IF: 3.5

2024-11-01

Royal Society Open Science

Abstract:One goal of Open Science is to promote reusability, which requires understanding and documentation. Reusability spans a spectrum from the straightforward reuse of existing materials to the extraction and adaptation of specific elements, depending on the maturity of the reused research and the research context. Beyond knowledge, understanding is crucial for enabling reusability. Simply reading an article is often insufficient; thus, publishing the underlying data and software is recommended. While the FAIR principles (Findable, Accessible, Interoperable, Reusable) facilitate the discovery and legal reuse of data and software through metadata, they fall short of promoting comprehensive understanding. By applying insights from the epistemology of simulation to computational mechanics research cases, we tested our hypothesis that the simulation process involves critical components beyond software and data that are essential to understand for reuse. Our findings indicate that reusability in these cases predominantly involves adjusting existing methodologies—a combination of different process steps from an epistemological perspective. Therefore, it is imperative to document not only learned knowledge but also the decisions regarding adjustments, assumptions and applications that concern the entire simulation process. Documentation is vital for understanding and herby enabling true reusability in scientific research, aligning with philosophical considerations of transparency and the nature of scientific knowledge.

multidisciplinary sciences
Using credible failure rates for SIS ensures process safety: A summary of the panel discussion

Julia V. Bukowski,Angela E. Summers,William M. Goble,Rajeev Limaye,Juan Gutierrez

DOI: https://doi.org/10.1002/prs.12489

2023-06-13

Process Safety Progress

Abstract:On March 13, 2023, in conjunction with the 19th Global Congress on Process Safety, four of the authors gathered to participate in a panel session. The discussion centered on the premise that many individual device failure rates published by some manufacturers and some certifying agencies are deemed—by members of the panel, many safety instrumented system (SIS) designers, and practicing engineers in the field—to be too low. This is especially true for mechanical final elements. A SIS designed using these unrealistic failure rates often results in estimated safety integrity levels (SILs) for the individual safety instrumented functions (SIFs) comprising the SIS, which would be achievable only under the most optimistic assumptions regarding installation, operation, and maintenance. These estimated SILs are unlikely to be achieved in the field. The panel discussed the underlying causes of this serious safety issue and proposed some possible solutions. This paper summarizes the panel discussion among four of the authors. Relevant graphics shown during the discussion and some panelists' quotations are included.

engineering, chemical

The importance of documenting failure

Creativity is Allowing for Failures: Leader Failure Tolerance and Employee Creativity

Success and Failure in Software Engineering: a Followup Systematic Literature Review

Illuminating 'the ugly side of science': fresh incentives for reporting negative results

Mid-career pitfall of consecutive success in science

Reflections on Software Failure Analysis

Designing for Reproducibility: A Qualitative Study of Challenges and Opportunities in High Energy Physics

The impact of failure on faculty development

Deceiving scientific research, misconduct events are possibly a more common practice than foreseen

Quantifying Success in Science: An Overview

When failure fails to be productive: probing the effectiveness of productive failure for learning beyond STEM domains

Teaching about the birth of synchrotron light: the role of Frascati and a missed opportunity

Documenting research software in engineering science

Negative equity - the value of reporting negative results

FAIL Is Not a Four-Letter Word: A Theoretical Framework for Exploring Undergraduate Students’ Approaches to Academic Challenge and Responses to Failure in STEM Learning Environments

Promoting and Supporting Innovation Through Collaborative Research

Reflections on Failure Education to Foster Creativity in the Age of Innovation

Why (and how) we should publish negative data

The Role of Failure in the Entrepreneurial Process: A Systematic Literature Review

Documenting research in simulation science to enhance understanding for reusability

Using credible failure rates for SIS ensures process safety: A summary of the panel discussion