Small effect size leads to reproducibility failure in resting-state fMRI studies
Xi-Ze Jia,Na Zhao,Barek Barton,Roxana G. Burciu,Nicolas Carriere,Antonio Cerasa,Bo-Yu Chen,Jun Chen,Stephen A. Coombes,Luc Defebvre,Christine Delmaire,Kathy Dujardin,Fabrizio Esposito,Guo-Guang Fan,Di Nardo Federica,Yi-Xuan Feng,Brett W. Fling,Saurabh Garg,Moran Gilat,Martin Gorges,Sl Ho,Fay B. Horak,Xiao Hu,Xiao-Fei Hu,Biao Huang,Pei-Yu Huang,Ze-Juan Jia,Christy Jones,Jan Kassubek,Lenka Krajčovičová,Ajay S. Kurani,Jing Li,Qian Li,Ai-Ping Liu,Bo Liu,Hu Liu,Wei-Guo Liu,Renaud Lopes,Yu-Ting Lou,Wei Luo,Tara M. Madhyastha,Ni-Ni Mao,Grainne M. McAlonan,Martin J. McKeown,Shirley Yy Pang,Aldo Quattrone,Irena Rektorová,Alessia Sarica,Hui-Fang Shang,James M. Shine,Priyank Shukla,Tomáš Slavíček,Xiao-Peng Song,Gioacchino Tedeschi,Alessandro Tessitore,David E. Vaillancourt,Jian Wang,Jue Wang,Z. Jane Wang,Lu-Qing Wei,Xia Wu,Xiao-Jun Xu,Lei Yan,Jing Yang,Wan-Qun Yang,Nai-Lin Yao,De-Long Zhang,Jiu-Quan Zhang,Min-Ming Zhang,Yan-Ling Zhang,Cai-Hong Zhou,Chao-Gan Yan,Xi-Nian Zuo,Mark Hallett,Tao Wu,Yufeng Zang
DOI: https://doi.org/10.1101/285171
2018-01-01
bioRxiv
Abstract:Thousands of papers using resting-state functional magnetic resonance imaging (RS-fMRI) have been published on brain disorders. Results in each paper may have survived correction for multiple comparison. However, since there have been no robust results from large scale meta-analysis, we do not know how many of published results are truly positives. The present meta-analytic work included 60 original studies, with 57 studies (4 datasets, 2266 participants) that used a between-group design and 3 studies (1 dataset, 107 participants) that employed a within-group design. To evaluate the effect size of brain disorders, a very large neuroimaging dataset ranging from neurological to psychiatric disorders together with healthy individuals have been analyzed. Parkinson9s disease off levodopa (PD-off) included 687 participants from 15 studies. PD on levodopa (PD-on) included 261 participants from 9 studies. Autism spectrum disorder (ASD) included 958 participants from 27 studies. The meta-analyses of a metric named amplitude of low frequency fluctuation (ALFF) showed that the effect size (Hedges9g) was 0.19 - 0.39 for the 4 datasets using between-group design and 0.46 for the dataset using within-group design. The effect size of PD-off, PD-on and ASD were 0.23, 0.39, and 0.19, respectively. Using the meta-analysis results as the robust results, the between-group design results of each study showed high false negative rates (median 99%), high false discovery rates (median 86%), and low accuracy (median 1%), regardless of whether stringent or liberal multiple comparison correction was used. The findings were similar for 4 RS-fMRI metrics including ALFF, regional homogeneity, and degree centrality, as well as for another widely used RS-fMRI metric namely seed-based functional connectivity. These observations suggest that multiple comparison correction does not control for false discoveries across multiple studies when the effect sizes are relatively small. Meta-analysis on un-thresholded t-maps is critical for the recovery of ground truth. We recommend that to achieve high reproducibility through meta-analysis, the neuroimaging research field should share raw data or, at minimum, provide un-thresholded statistical images.