Abstract:Introduction: Resting-state functional magnetic resonance imaging (RfMRI) has been employed to investigate brain functional changes in a variety of neurological and psychiatric disorders. Understanding the reliability of this technique is essential to its successful application. Numerous studies have focused on the test-retest reliability (i.e., reliability in repeated measurements using the same cohort). However, studies focusing on the between-cohort reliability are rare. Between-cohort reliability answers the important question of whether the biomarker developed using a limited number of subjects can be applied to the general population. This study will demonstrate the key factors that affect the reliability of detecting functional connectivity changes in studying brain disorders. Methods: RfMRI data from 65 Alzheimer's disease (AD) and 136 cognitively normal (CN) subjects were included in the study. Data for this work were from the Medical College of Wisconsin (MCW) fMRI Data Bank and the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (adni.loni.ucla.edu). Experiment 1. Seed-based analysis was employed in this study. Six seed ROIs (left and right hippocampus, posterior cingulate cortex (PCC) and Insula) were chosen, based on prior knowledge in the literature (1–3). The subjects were randomly assigned to a test and validation cohort. The test cohort was employed to define the RfMRI biomarker. The validation cohort was used to validate the biomarker (i.e., functional connectivity index, FCI). When the FCIs were significantly different between the validation AD and CN cohorts, the biomarker is considered reliable and vice versa. The above steps were repeated 100 times. We defined the RfMRI reliabilities as the rate of validating the RfMRI biomarker (calculated by dividing the number of reliable outcomes by the number of repetitions). Fig. 1 shows the detailed procedure. This procedure was performed 10 times to obtain the mean and standard deviation of the reliability. To study the effect of the sample size on the reliability, different numbers of subjects in each (test, validation) cohort, ranging from eight to 100, were employed. Experiment 2. To compare the effects of between-subject variation and between-scan variation, the resting-state data of the testing cohort (used in Experiment 1) were divided to two parts (first and second half of the scans). In Experiment 2, the first half of resting state data was used as the testing data and the second half of the data was used as the validation data. The reliability was then obtained the same way, as described in Experiment 1. Experiment 3. To characterize the random effect (i.e., false-positive reliability), pseudo groups were created. The pseudo group (pseudo test AD group, for example) contained CN and AD subjects. The "reliability" was then calculated using the pseudo groups.

Impact of analytic decisions on test-retest reliability of individual and group estimates in functional magnetic resonance imaging: a multiverse analysis using the monetary incentive delay task

Influences on the Test–Retest Reliability of Functional Connectivity MRI and its Relationship with Behavioral Utility

Small effect size leads to reproducibility failure in resting-state fMRI studies

Data-analytical stability of cluster-wise and peak-wise inference in fMRI data analysis

Modified test statistics by inter-voxel variance shrinkage with an application to f MRI.

How do tasks impact the reliability of fMRI functional connectivity?

Reliability of Detecting Functional Connectivity Changes in Alzheimer ' s Disease

Impact of Independent Component Analysis Dimensionality on the Test-Retest Reliability of Resting-State Functional Connectivity

Influence of sample size and analytic approach on stability and interpretation of brain‐behavior correlations in task‐related fMRI data

Complementary benefits of multivariate and hierarchical models for identifying individual differences in cognitive control

Standardizing the intrinsic brain: Towards robust measurement of inter-individual variation in 1000 functional connectomes.

MRI economics: Balancing sample size and scan duration in brain wide association studies

Test–retest reliability of approach‐avoidance conflict decision‐making during functional magnetic resonance imaging in healthy adults

Assessing Variations in Areal Organization for the Intrinsic Brain: from Fingerprints to Reliability

Reliability and reproducibility of individual differences in functional connectivity acquired during task and resting state

Assessment and Optimization of Functional MRI Analyses

Comparison of oxyhemoglobin and deoxyhemoglobin signal reliability with and without global mean removal for digit manipulation motor tasks

Within-subject variation in BOLD-fMRI signal changes across repeated measurements: quantification and implications for sample size.

Stability and Localization of inter-individual differences in functional connectivity

A decade of test-retest reliability of functional connectivity: A systematic review and meta-analysis

When most fMRI connectivity cannot be detected: insights from time course reliability