The Preprocessed Connectomes Project Quality Assessment Protocol - a Resource for Measuring the Quality of MRI Data.
Shehzad Zarrar,Giavasis Steven,Li Qingyang,Benhajali Yassine,Yan Chaogan,Yang Zhen,Milham Michael,Bellec Pierre,Craddock Cameron
DOI: https://doi.org/10.3389/conf.fnins.2015.91.00047
IF: 4.3
2015-01-01
Frontiers in Neuroscience
Abstract:Event Abstract Back to Event The Preprocessed Connectomes Project Quality Assessment Protocol - a resource for measuring the quality of MRI data. Zarrar Shehzad1, 2, 3, Steven Giavasis1, 2, Qingyang Li1, Yassine Benhajali4, 5, Chaogan Yan1, 2, Zhen Yang1, Michael Milham1, 2, Pierre Bellec5 and Cameron Craddock1, 2* 1 Child Mind Institute, Center for the Developing Brain, United States 2 Nathan S. Kline Institute for Psychiatric Research, Center for Biomedical Imaging and Neuromodulation, United States 3 Yale University, Department of Psychology, United States 4 Université de Montréal, Dépt d'anthropologie, Canada 5 Centre de recherche de l'institut de gériatrie de Montréal, Canada Background: Although several measures have been proposed for assessing the quality of structural and fMRI data, there is no clear guidance on which of the measures are the best indicators of quality, or on the ranges of values that constitute "good" or "bad" data. This is particularly problematic for resting state fMRI (R-fMRI) data, since there is no clear means for differentiating signal from noise. As a result, researchers are required to rely on painstaking visual inspection to assess data quality. But this approach consumes a lot of time and resources, is subjective, and is susceptible to inter-rater and test-retest variability. Additionally, it is possible that some defects are too subtle to be fully appreciated by visual inspection, yet are strong enough to degrade the accuracy of data processing algorithms or bias analysis results. Further, it is very difficult to visually assess the quality of data that has already been processed, such as that being shared through the Preprocessed Connectomes Project (PCP; http://preprocessed-connectomes-project.github.io/). To begin to address this problem, the PCP has assembled several of the quality metrics proposed in the literature to implement a Quality Assessment Protocol (QAP; http://preprocessed-connectomes-project.github.io/quality-assessment-protocol). The QAP includes measures for assessing the quality of both functional and structural MRI data. The quality of structural MRI data is assessed using contrast-to-noise ratio (CNR; Magnotta and Friedman, 2006), entropy focus criterion (EFC, Atkinson 1997), foreground-to-background energy ratio (FBER), voxel smoothness (FWHM, Friedman 2008), percentage of artifact voxels (QI1, Mortamet 2009), and signal-to-noise ratio (SNR, Magnotta and Friedman, 2006). The QAP includes methods to assess both the spatial and temporal quality of fMRI data. Spatial quality is assessed using EFC, FBER, and FWHM, in addition to ghost-to-signal ratio (GSR). Spatial quality metrics were calculated for functional data using the mean image. Temporal quality of functional data is assessed using the standardized root mean squared change in fMRI signal between volumes (DVARS; Nichols 2013), mean root mean square deviation (MeanFD, Jenkinson 2003), the percentage of voxels with meanFD > 0.2 (Percent FD; Power 2012), the temporal mean of AFNI's 3dTqual metric (1 minus the Spearman correlation between each fMRI volume and the median volume; Cox 1995) and the average fraction of outliers found in each volume using AFNI's 3dTout command. To build normative distributions of data quality we applied the QAP Python toolbox (https://github.com/preprocessed-connectomes-project/quality-assessment-protocol) to measure structural and temporal data quality on data from the Autism Brain Imaging Data Exchange (ABIDE; Di Martino 2013) and the Consortium for Reliability and Reproducibility (CoRR, Zuo 2014). We further analyzed the properties of QAP measures, by analyzing the calculated measures to evaluate their collinearity, correspondence to expert-assigned quality labels, and test-retest reliability. Methods: The QAP python toolbox was used to calculate spatial and temporal quality measures on the 1,113 structural and functional MRI datasets from the ABIDE dataset and the 3,357 structural and 5,094 functional scans from the CoRR dataset. For the ABIDE data, quality measures were compared to the quality scores determined from visual inspection by three expert raters to evaluate their predictive value. For both the ABIDE and CoRR datasets, the redundancy between quality measures was evaluated from their correlation matrix. Finally, the test-retest reliability of quality measures derived from CoRR was assessed using intra-class correlation. Results: Each of the measures showed a good bit of variability between imaging sites (see Figure 1 for an example plot showing standardized DVARS for ABIDE). Ranks calculated from the weighted average of standardized quality metrics indicated that CMU was the worst performing site and NYU was the best. QI1 and SNR were the best predictors of manually applied structural data quality scores, and EFC, FWHM, Percent FD, and GSR were all significant predictors of functional data quality (fig 2, p<0.0001). A few of the measures are highly correlated (fig. 3) such as SNR, CNR and FBER, which measure very similar constructs, indicated that there is some room for reducing the set of measures. For the functional data, the test-retest reliability of several of the spatial measures of quality were very high (fig 4., EFC, FBER, GSR) reflecting their sensitivity to technical quality (i.e. MR system and parameters) whereas temporal measures were lower reflecting their sensitivity to physiological factors such as head motion. Similarly in the structural data, it appears that measures can be divided into those that are more sensitive to technical quality (EFC, FWHM) and those that favor physiological variation (CNR, QI1) based on test-retest reliability. Conclusions: We have assembled a diverse set of QA metrics for assessing the quality of R-fMRI data. The resulting Python toolbox was used to build distributions of the metrics for the ABIDE and CoRR datasets that can be used as a standard for comparing the quality of other datasets and eventually devising algorithms for automated QA. It appears as though test-retest reliability of the different measures can help distinguish those that are more sensitive to technical variation from those that are sensitive to physiology. Figure 1 Figure 2 Figure 3 Figure 4 References Magnotta, V. A., & Friedman, L. (2006). Measurement of signal-to-noise and contrast-to-noise in the fBIRN multicenter imaging study. Journal of Digital Imaging, 19(2), 140-147. Atkinson D, Hill DL, Stoyle PN, Summers PE, Keevil SF (1997). Automatic correction of motion artifacts in magnetic resonance images using an entropy focus criterion. IEEE Trans Med Imaging. 16(6):903-10. Friedman, L., Stern, H., Brown, G. G., Mathalon, D. H., Turner, J., Glover, G. H., … & Potkin, S. G. (2008). Test–retest and between‐site reliability in a multicenter fMRI study. Human brain mapping, 29(8), 958-972. ↩ Mortamet, B., Bernstein, M. A., Jack, C. R., Gunter, J. L., Ward, C., Britson, P. J., … & Krueger, G. (2009). Automatic quality assessment in structural brain magnetic resonance imaging. Magnetic Resonance in Medicine, 62(2), 365-372. Power, J. D., Barnes, K. A., Snyder, A. Z., Schlaggar, B. L. & Petersen, S. E. (2012) Spurious but systematic correlations in functional connectivity MRI networks arise from subject motion. Neuroimage 59, 2142-2154. Nichols, T. (2012, Oct 28). Standardizing DVARS. Retrieved from http://blogs.warwick.ac.uk/nichols/entry/standardizing_dvars. Cox, R.W. (1996) AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages. Computers and Biomedical Research, 29:162-173. Jenkinson, M., Bannister, P., Brady, M., & Smith, S. (2002). Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage, 17(2), 825-841. Yan CG, Cheung B, Kelly C, Colcombe S, Craddock RC, Di Martino A, Li Q, Zuo XN, Castellanos FX, Milham MP (2013). A comprehensive assessment of regional variation in the impact of head micromovements on functional connectomics. Neuroimage. 76:183-201. Keywords: MRI, fMRI methods, resting state fMRI, Quality control, connectomes, python, tools, data sharing Conference: Neuroinformatics 2015, Cairns, Australia, 20 Aug - 22 Aug, 2015. Presentation Type: Poster, to be considered for oral presentation Topic: Neuroimaging Citation: Shehzad Z, Giavasis S, Li Q, Benhajali Y, Yan C, Yang Z, Milham M, Bellec P and Craddock C (2015). The Preprocessed Connectomes Project Quality Assessment Protocol - a resource for measuring the quality of MRI data.. Front. Neurosci. Conference Abstract: Neuroinformatics 2015. doi: 10.3389/conf.fnins.2015.91.00047 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers' terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 08 Apr 2015; Published Online: 05 Aug 2015. * Correspondence: Dr. Cameron Craddock, Child Mind Institute, Center for the Developing Brain, New York, New York, 10022, United States, cameron.craddock@austin.utexas.edu Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Zarrar Shehzad Steven Giavasis Qingyang Li Yassine Benhajali Chaogan Yan Zhen Yang Michael Milham Pierre Bellec Cameron Craddock Google Zarrar Shehzad Steven Giavasis Qingyang Li Yassine Benhajali Chaogan Yan Zhen Yang Michael Milham Pierre Bellec Cameron Craddock Google Scholar Zarrar Shehzad Steven Giavasis Qingyang Li Yassine Benhajali Chaogan Yan Zhen Yang Michael Milham Pierre Bellec Cameron Craddock PubMed Zarrar Shehzad Steven Giavasis Qingyang Li Yassine Benhajali Chaogan Yan Zhen Yang Michael Milham Pierre Bellec Cameron Craddock Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.