Abstract:The final, formatted version of the article will be published soon. You have multiple emails registered with Frontiers: Please enter your email address: If you already have an account, please login You don't have a Frontiers account ? You can register here After more than a decade since the Imagenet breakthrough (Krizhevsky et al., 2012), there is no doubt that Deep Learning (DL) has established itself as a powerful resource whose limits and risks remain difficult to assess (Bengio et al., 2024). This success, originated from the ability of deep neural networks to create representations of complex data with multiple levels of abstraction (LeCun et al., 2015), has inevitably attracted researchers from different domains, including multidisciplinary ones.In computational neuroscience (Trappenberg, 2009), deep learning has offered novel insights into the functionalities of the brain, demonstrating a remarkable ability to exploit the intrinsic multimodality of the field (Saxe et al., 2021). This is also reflected in the increasing volume of publications encompassing diverse data types such as electroencephalographam (EEG), magnetoencephalographam (MEG), structural (MRI) and functional magnetic resonance imaging (fMRI) (Zhu et al., 2019;Zhang et al., 2021). Nevertheless, the potential of deep learning in neuroimaging data analysis is countered by several critical issues (Miotto et al., 2018), including the scarcity of large open datasets, the poor generalizability of DL models, their lack of interpretability, and the poor reproducibility of results, which is discussed in this work.According to (National Academies of Sciences, Engineering, and Medicine et al., 2019), reproducibility is defined as the ability to "obtain consistent results using the same input data; computational steps, methods, and code; and conditions of analysis." This definition differs from that of replicability, commonly mistaken as a synonym, which is instead defined as the ability to "obtain consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data." From the above definitions, it is possible to derive how reproducibility solely depends on how authors facilitate the emulation of the same computational environment. Improving reproducibility, especially in a deep learning scenario, is therefore crucial for ensuring methodological robustness and result trustworthiness. However, when analyzing neuroimaging deep learning studies, several researchers have expressed concerns on how authors rarely report the key elements that make a study reproducible (Ciobanu-Caraus et al., 2024;Colliot et al., 2023). A recent review of DL applications for medical image segmentation found that only 9% of the selected studies were reproducible (Renard et al., 2020), a conclusion also supported by other independent studies (Moassefi et al., 2024;Marrone et al., 2019;Ligneris et al., 2023). Furthermore, the same problem was discovered in DL-EEG applications, where, from a review of 154 selected papers, only 12 were found to be easily reproducible (Roy et al., 2019).The above statistics highlight not only the severity of the reproducibility crisis, but also how often this important issue is overlooked by both authors and publishers. Insufficient reproducibility not only threatens the credibility of scientific findings, potentially hindering the discovery of new knowledge in the domain (e.g., treatments for neurological disorders), but also introduces inconsistencies in results due to factors such as dataset diversity, variability in preprocessing, and discrepancies in model implementation and evaluation. Such inconsistencies heighten the risk of misinterpreting data or drawing incorrect conclusions that support the validity of a framework over another, thereby posing a potential negative impact on clinical outcomes. For instance, various studies using deep learning to classify different types of dementia or to predict cognitive scores with extremely high accuracies have been questioned not only for their lack of reproducibility, but also for potential performance biases arising from inadequate data partitioning methods or ambiguous validation or model selection procedures (Brookshire et al., 2024;Wen et al., 2020). In contrast, other disciplines have achieved greater reproducibility through the use of standardized datasets and methodologies. For example, the ImageNet dataset (Krizhevsky et al., 2012) has significantly advanced computer vision by establishing benchmarks categorized by learning methods, while the General Language Understanding Evaluation (GLUE) benchmark provides a collection of resources for training, evaluating, and analyzing natural language understanding systems (Wang et al., 2018).Consequently, there is a need for the neuroimaging field to adopt similar practices to improve the reliability of published -Abstract Truncated-

Reporting checklists in neuroimaging: promoting transparency, replicability, and reproducibility

Scanning the horizon: towards transparent and reproducible neuroimaging research

CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII

Ten simple rules for neuroimaging meta-analysis

Open and reproducible neuroimaging: From study inception to publication

Updating the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) for reporting AI research

Reproducible brain PET data analysis: easier said than done

A Checklist for Assessing the Methodological Quality of Concurrent Tes-Fmri Studies (contes Checklist): a Consensus Study and Statement

Lessons learned: A neuroimaging research center's transition to open and reproducible science

A methodological checklist for fMRI drug cue reactivity studies: development and expert consensus

Toward improving reproducibility in neuroimaging deep learning studies

Guidelines for the content and format of PET brain data in publications and archives: A consensus paper

A Standardized Protocol for Efficient and Reliable Quality Control of Brain Registration in Functional MRI Studies

CLEAR guideline for radiomics: Early insights into current reporting practices endorsed by EuSoMII

Assessing Methods and Tools to Improve Reporting, Increase Transparency, and Reduce Failures in Machine Learning Applications in Health Care

Community-developed checklists for publishing images and image analysis

Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.

Evaluation of the reliability and usability of CARE‐Radiology: A descriptive‐analytic study

Checklist As a Memory Externalization Tool During a Critical Care Process.

A Preliminary Checklist (METRICS) to Standardize the Design and Reporting of Studies on Generative Artificial Intelligence–Based Models in Health Care Education and Practice: Development Study Involving a Literature Review