Abstract:It is our pleasure to see the comments from Dr. Serge Koscielny and Stefan Michiels regarding our re-evaluation (1) of their study (2). We agree that the current challenge in developing predictive models using clinical genomics data is not to construct a statistically robust model, but in developing a model with the capability and accuracy for real-world applications such as, in this case, cancer prognosis. Many factors, including gene selection rules, sample resubstitution approaches, sample size concerns, classification methods, and validation strategies, contribute to the variation in microarray-based models (3–6). We believe that by exploring this vast parameter space, one can find what appears to be a good model as demonstrated by internal validation, if he tries hard enough. Thus, we stressed in our study that “caution should be taken when a general conclusion is made on the basis of a single statistical practice without alternative validation”.Thinking retrospectively, many published studies, including that by Michiels et al. (2) and our reanalysis (1), draw conclusions based on the analysis of one population of samples that is split randomly into two, one for training and the other for validation. This commonly adopted strategy does not represent clinical medicine well, where a diagnostic test needs to be validated in a different population than the one that is used to develop the test. This may explain why many gene signatures, even those with significant prediction accuracy over chance, are then rejected in independent validation studies. An improved strategy would include validation using only those patients whose genomic profiles and clinical assessment were done at a different time period (preferably at a later date) as compared with those used in the training set.Fortunately, the Food and Drug Administration (FDA)-led, community-wide MicroArray Quality Control (MAQC) project applied this strategy of generating validation samples profiled at a later date than training samples in its second phase of study, aiming at establishing best practices for the use of microarrays in real-world scenarios by comparing different approaches to the development and validation of microarray-based models (7). Over 30,000 models for 13 endpoints from 6 relatively large data sets (including 3 clinical data sets: breast cancer, multiple myeloma, and neuroblastoma) were generated by 36 teams. The main conclusions of this large-scale study were summarized in over 10 manuscripts that were published in Nature Biotechnology and The Pharmacogenomics Journal in August, 2010. This effort will advance our knowledge on the best ways to validate clinical genomics signatures.See the original Letter to the Editor, p. 6180The views presented in this article do not necessarily reflect those of the U.S. Food and Drug Administration.

Reproducible and Reliable Microarray Results Through Quality Control: Good Laboratory Proficiency and Appropriate Data Analysis Practices Are Essential

QA/QC Issues to Aid Regulatory Acceptance of Microarray Gene Expression Data

QA/QC: challenges and pitfalls facing the microarray community and regulatory agencies.

The Microarray Quality Control (Maqc) Project And Cross-Platform Analysis Of Microarray Data

The Application of Novel Multifunctional Microarray Slides for Immobilization Biomolecules

Microarray Technology: Unresolved Issues And Future Challenges From A Regulatory Perspective

The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements

Technical Issues Involved in Obtaining Reliable Data from Microarray Experiments

Standardization efforts enabling next-generation sequencing and microarray based biomarkers for precision medicine.

Cross-platform comparability of microarray technology: Intra-platform consistency and appropriate data analysis procedures are essential

Approaches and Practical Considerations for the Analysis of Toxicogenomics Data

Quality assurance of RNA expression profiling in clinical laboratories

Clinical Utility of Microarrays: Current Status, Existing Challenges and Future Outlook

Statistical Evaluation of Clinical Usefulness of Microarrays for Cancer Prognosis Needs to Be Placed in the Context of Clinical Reality-Response

Dna Microarrays: Applications, Future Trends, And The Need For Standardization

Investigation of reproducibility of differentially expressed genes in DNA microarrays through statistical simulation

Improvement in the Reproducibility and Accuracy of DNA Microarray Quantification by Optimizing Hybridization Conditions

Cross-platform comparison of SYBR ® Green real-time PCR with TaqMan PCR, microarrays and other gene expression measurement technologies evaluated in the MicroArray Quality Control (MAQC) study

Quality Assessment for Short Oligonucleotide Microarray Data

The Current Status of Dna Microarrays

Development of Public Toxicogenomics Software for Microarray Data Management and Analysis