Abstract:It is our pleasure to see the comments from Dr. Serge Koscielny and Stefan Michiels regarding our re-evaluation (1) of their study (2). We agree that the current challenge in developing predictive models using clinical genomics data is not to construct a statistically robust model, but in developing a model with the capability and accuracy for real-world applications such as, in this case, cancer prognosis. Many factors, including gene selection rules, sample resubstitution approaches, sample size concerns, classification methods, and validation strategies, contribute to the variation in microarray-based models (3–6). We believe that by exploring this vast parameter space, one can find what appears to be a good model as demonstrated by internal validation, if he tries hard enough. Thus, we stressed in our study that “caution should be taken when a general conclusion is made on the basis of a single statistical practice without alternative validation”.Thinking retrospectively, many published studies, including that by Michiels et al. (2) and our reanalysis (1), draw conclusions based on the analysis of one population of samples that is split randomly into two, one for training and the other for validation. This commonly adopted strategy does not represent clinical medicine well, where a diagnostic test needs to be validated in a different population than the one that is used to develop the test. This may explain why many gene signatures, even those with significant prediction accuracy over chance, are then rejected in independent validation studies. An improved strategy would include validation using only those patients whose genomic profiles and clinical assessment were done at a different time period (preferably at a later date) as compared with those used in the training set.Fortunately, the Food and Drug Administration (FDA)-led, community-wide MicroArray Quality Control (MAQC) project applied this strategy of generating validation samples profiled at a later date than training samples in its second phase of study, aiming at establishing best practices for the use of microarrays in real-world scenarios by comparing different approaches to the development and validation of microarray-based models (7). Over 30,000 models for 13 endpoints from 6 relatively large data sets (including 3 clinical data sets: breast cancer, multiple myeloma, and neuroblastoma) were generated by 36 teams. The main conclusions of this large-scale study were summarized in over 10 manuscripts that were published in Nature Biotechnology and The Pharmacogenomics Journal in August, 2010. This effort will advance our knowledge on the best ways to validate clinical genomics signatures.See the original Letter to the Editor, p. 6180The views presented in this article do not necessarily reflect those of the U.S. Food and Drug Administration.

Functional analysis of multiple genomic signatures demonstrates that classification algorithms choose phenotype-related genes

Signature Genes Selection and Functional Analysis of Phenotypes: A Comparative Study

Common Human Cancer Genes Discovered by Integrated Gene-Expression Analysis

Effect of training-sample size and classification difficulty on the accuracy of genomic predictors

Derivation of stable microarray cancer-differentiating signatures using consensus scoring of multiple random sampling and gene-ranking consistency evaluation.

Mining gene expression data of multiple sclerosis

Maximizing Biomarker Discovery by Minimizing Gene Signatures

Identification and Validation of Prognostically Relevant Gene Signature in Melanoma

Consistent metagenes from cancer expression profiles yield agent specific predictors of chemotherapy response

A two step method to identify clinical outcome relevant genes with microarray data.

An integral genomic signature approach for tailored cancer therapy using genome-wide sequencing data

Identifying Anti-Cancer Drug Response Related Genes Using an Integrative Analysis of Transcriptomic and Genomic Variations with Cell Line-Based Drug Perturbations.

Comparison of RNA-seq and microarray-based models for clinical endpoint prediction

Towards precise classification of cancers based on robust gene functional expression profiles

Maximum predictive power of the microarray-based models for clinical outcomes is limited by correlation between endpoint and gene expression profile

Toxicogenomic module associations with pathogenesis: a network-based approach to understanding drug toxicity

Disease-specific classification using deconvoluted whole blood gene expression

Statistical Evaluation of Clinical Usefulness of Microarrays for Cancer Prognosis Needs to Be Placed in the Context of Clinical Reality-Response

An investigation of biomarkers derived from legacy microarray data for their utility in the RNA-seq era

Identifying Disease Feature Genes Based on Cellular Localized Gene Functional Modules and Regulation Networks

Application of genomics in preclinical drug safety evaluation.