Abstract:Publications proposing novel machine learning methods are often primarily rated by exhibited predictive performance on selected problems. In this position paper we argue that predictive performance alone is not a good indicator for the worth of a publication. Using it as such even fosters problems like inefficiencies of the machine learning research community as a whole and setting wrong incentives for researchers. We therefore put out a call for the publication of "negative" results, which can help alleviate some of these problems and improve the scientific output of the machine learning research community. To substantiate our position, we present the advantages of publishing negative results and provide concrete measures for the community to move towards a paradigm where their publication is normalized.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to explore and solve the problem of over - relying on predictive performance in the field of machine - learning research. Specifically, the author believes that there are several main problems in current machine - learning research: 1. **Limitations of a single performance metric**: - The paper points out that many current machine - learning studies rely too much on predictive performance to evaluate the value of new methods. This approach not only ignores other important evaluation criteria (such as novelty, theoretical contributions, etc.), but may also lead to a waste of research resources and duplication of efforts. - Definition 1.1: In empirical machine - learning, the usual null hypothesis is that the proposed method does not significantly outperform the existing methods in terms of predictive performance on the relevant subset of problems. 2. **The importance of negative results is ignored**: - If a new method or algorithm cannot outperform existing methods on typical benchmark datasets, researchers may quickly abandon their work because such results are difficult to publish. However, these so - called "negative results" can actually provide valuable information and inspiration for subsequent research. - Definition 1.2: When the usual null hypothesis cannot be rejected, that is, when the new method is considered not to be better than the existing method, this is a negative result in empirical machine - learning. 3. **Inefficiency in the research community and poor incentive mechanisms**: - Excessive focus on predictive performance has led to inefficiency in the research community. Many researchers tend to choose research directions that are likely to produce positive results in order to pursue publishable results, while ignoring innovative research that may be more meaningful but riskier. - Researchers are motivated to pursue work that can demonstrate performance improvements rather than re - implementing and validating existing methods, which further exacerbates the inequality in resource allocation and the inefficiency of research. 4. **Publication Bias**: - Publication bias means that only those studies that show significant improvements are more likely to be accepted for publication, while negative results are often ignored. This phenomenon can lead to biases in scientific research, preventing some valuable but ineffective research results from being made public and affecting the overall progress of science. ### Solutions To solve the above problems, the author calls on the academic community to value and encourage the publication of negative results. Specific measures include: - **Emphasizing the value of negative results**: By publishing negative results, other researchers can be helped to avoid repeating ineffective work and gain new insights and improvements from it. - **Changing the review criteria**: Reviewers should consider multiple factors comprehensively, not just predictive performance, to more comprehensively evaluate the value of research work. - **Establishing appropriate incentive mechanisms**: Encourage researchers to conduct more extensive and in - depth research, even if these studies ultimately fail to achieve the expected performance improvements. - **Promoting transparency and reproducibility**: By recording and sharing experimental processes and results (including negative results) in detail, the transparency and reproducibility of research can be improved, thereby promoting the healthy development of the entire field. In short, this paper hopes to improve the efficiency and quality of the machine - learning research community, reduce publication bias, and provide a more comprehensive and reliable foundation for future scientific research work by advocating the publication of negative results.

Position: Embracing Negative Results in Machine Learning

Illuminating 'the ugly side of science': fresh incentives for reporting negative results

Nil nisi bene?

Why (and how) we should publish negative data

Negative equity - the value of reporting negative results

Perspectives on Negative Research Results in Pervasive Computing

So you got a null result. Will anyone publish it?

Exploring the Scientific Impact of Negative Results

A critical scientific evaluation of a purportedly negative data report – response to Seneviratne et al. 2022

Loss Aversion in Recommender Systems: Utilizing Negative User Preference to Improve Recommendation Quality

Animal researchers' views on the publication of negative results and subsequent policy adoptions

Position Paper: Rethinking Empirical Research in Machine Learning: Addressing Epistemic and Methodological Challenges of Experimentation

Rethinking negative sampling in content-based news recommendation

Does Negative Sampling Matter? A Review with Insights into its Theory and Applications

Statistical Thinking, Machine Learning

Positive thinking about negative studies

On the Role of Negative Precedent in Legal Outcome Prediction

AI & robotics briefing: Why AI needs to see the 'ugly' side of science

It's difficult, but important, to make negative predictions

When is Nothing Something? Editorial for the Null Results Special Issue of Journal of Business and Psychology

Negative Sampling in Recommendation: A Survey and Future Directions