Statistical Evaluation of Clinical Usefulness of Microarrays for Cancer Prognosis Needs to Be Placed in the Context of Clinical Reality-Response
Weida Tong,Xiaohui Fan
DOI: https://doi.org/10.1158/1078-0432.ccr-10-1921
IF: 13.801
2010-01-01
Clinical Cancer Research
Abstract:It is our pleasure to see the comments from Dr. Serge Koscielny and Stefan Michiels regarding our re-evaluation (1) of their study (2). We agree that the current challenge in developing predictive models using clinical genomics data is not to construct a statistically robust model, but in developing a model with the capability and accuracy for real-world applications such as, in this case, cancer prognosis. Many factors, including gene selection rules, sample resubstitution approaches, sample size concerns, classification methods, and validation strategies, contribute to the variation in microarray-based models (3–6). We believe that by exploring this vast parameter space, one can find what appears to be a good model as demonstrated by internal validation, if he tries hard enough. Thus, we stressed in our study that “caution should be taken when a general conclusion is made on the basis of a single statistical practice without alternative validation”.Thinking retrospectively, many published studies, including that by Michiels et al. (2) and our reanalysis (1), draw conclusions based on the analysis of one population of samples that is split randomly into two, one for training and the other for validation. This commonly adopted strategy does not represent clinical medicine well, where a diagnostic test needs to be validated in a different population than the one that is used to develop the test. This may explain why many gene signatures, even those with significant prediction accuracy over chance, are then rejected in independent validation studies. An improved strategy would include validation using only those patients whose genomic profiles and clinical assessment were done at a different time period (preferably at a later date) as compared with those used in the training set.Fortunately, the Food and Drug Administration (FDA)-led, community-wide MicroArray Quality Control (MAQC) project applied this strategy of generating validation samples profiled at a later date than training samples in its second phase of study, aiming at establishing best practices for the use of microarrays in real-world scenarios by comparing different approaches to the development and validation of microarray-based models (7). Over 30,000 models for 13 endpoints from 6 relatively large data sets (including 3 clinical data sets: breast cancer, multiple myeloma, and neuroblastoma) were generated by 36 teams. The main conclusions of this large-scale study were summarized in over 10 manuscripts that were published in Nature Biotechnology and The Pharmacogenomics Journal in August, 2010. This effort will advance our knowledge on the best ways to validate clinical genomics signatures.See the original Letter to the Editor, p. 6180The views presented in this article do not necessarily reflect those of the U.S. Food and Drug Administration.