Abstract:It is our pleasure to see the comments from Dr. Serge Koscielny and Stefan Michiels regarding our re-evaluation (1) of their study (2). We agree that the current challenge in developing predictive models using clinical genomics data is not to construct a statistically robust model, but in developing a model with the capability and accuracy for real-world applications such as, in this case, cancer prognosis. Many factors, including gene selection rules, sample resubstitution approaches, sample size concerns, classification methods, and validation strategies, contribute to the variation in microarray-based models (3–6). We believe that by exploring this vast parameter space, one can find what appears to be a good model as demonstrated by internal validation, if he tries hard enough. Thus, we stressed in our study that “caution should be taken when a general conclusion is made on the basis of a single statistical practice without alternative validation”.Thinking retrospectively, many published studies, including that by Michiels et al. (2) and our reanalysis (1), draw conclusions based on the analysis of one population of samples that is split randomly into two, one for training and the other for validation. This commonly adopted strategy does not represent clinical medicine well, where a diagnostic test needs to be validated in a different population than the one that is used to develop the test. This may explain why many gene signatures, even those with significant prediction accuracy over chance, are then rejected in independent validation studies. An improved strategy would include validation using only those patients whose genomic profiles and clinical assessment were done at a different time period (preferably at a later date) as compared with those used in the training set.Fortunately, the Food and Drug Administration (FDA)-led, community-wide MicroArray Quality Control (MAQC) project applied this strategy of generating validation samples profiled at a later date than training samples in its second phase of study, aiming at establishing best practices for the use of microarrays in real-world scenarios by comparing different approaches to the development and validation of microarray-based models (7). Over 30,000 models for 13 endpoints from 6 relatively large data sets (including 3 clinical data sets: breast cancer, multiple myeloma, and neuroblastoma) were generated by 36 teams. The main conclusions of this large-scale study were summarized in over 10 manuscripts that were published in Nature Biotechnology and The Pharmacogenomics Journal in August, 2010. This effort will advance our knowledge on the best ways to validate clinical genomics signatures.See the original Letter to the Editor, p. 6180The views presented in this article do not necessarily reflect those of the U.S. Food and Drug Administration.

Development and Validation of Predictive Molecular Signatures

Gaining Confidence on Molecular Classification Through Consensus Modeling and Validation

An integrated bioinformatics approach to early diagnosis, prognosis and therapeutics of non-small-cell lung cancer

A Detailed Catalogue of Multi-Omics Methodologies for Identification of Putative Biomarkers and Causal Molecular Networks in Translational Cancer Research

Reproducible and Reliable Microarray Results Through Quality Control: Good Laboratory Proficiency and Appropriate Data Analysis Practices Are Essential

Approaches to Uncovering Cancer Diagnostic and Prognostic Molecular Signatures.

Trials and tribulations of 'omics data analysis: assessing quality of SIMCA-based multivariate models using examples from pulmonary medicine

Bayesian profiling of molecular signatures to predict event times

Gene signatures for cancer research: A 25-year retrospective and future avenues

Statistical Evaluation of Clinical Usefulness of Microarrays for Cancer Prognosis Needs to Be Placed in the Context of Clinical Reality-Response

Public data and open source tools for multi-assay genomic investigation of disease

Molecular Signatures from Gene Expression Data

Signature Genes Selection and Functional Analysis of Phenotypes: A Comparative Study

From Data to Cure: A Comprehensive Exploration of Multi-omics Data Analysis for Targeted Therapies

Predicting Prognostic Bidirectional Molecular Signatures Associating Myocardial Infarction and Lung Cancer: An In-Silico Perspective

Prognostic Cancer Gene Expression Signatures: Current Status and Challenges

Selecting a single model or combining multiple models for microarray-based classifier development? – A comparative analysis based on large and diverse datasets generated from the MAQC-II project

The Microarray Quality Control (Maqc) Project And Cross-Platform Analysis Of Microarray Data

Identifying temporal molecular signatures underlying cardiovascular diseases: A data science platform

Standardization efforts enabling next-generation sequencing and microarray based biomarkers for precision medicine.

A novel approach for biomarker selection and the integration of repeated measures experiments from two assays