Abstract:Background Genomic biomarkers play an increasing role in both preclinical and clinical application. Development of genomic biomarkers with microarrays is an area of intensive investigation. However, despite sustained and continuing effort, developing microarray-based predictive models (i.e., genomics biomarkers) capable of reliable prediction for an observed or measured outcome (i.e., endpoint) of unknown samples in preclinical and clinical practice remains a considerable challenge. No straightforward guidelines exist for selecting a single model that will perform best when presented with unknown samples. In the second phase of the MicroArray Quality Control (MAQC-II) project, 36 analysis teams produced a large number of models for 13 preclinical and clinical endpoints. Before external validation was performed, each team nominated one model per endpoint (referred to here as 'nominated models') from which MAQC-II experts selected 13 'candidate models' to represent the best model for each endpoint. Both the nominated and candidate models from MAQC-II provide benchmarks to assess other methodologies for developing microarray-based predictive models. Methods We developed a simple ensemble method by taking a number of the top performing models from cross-validation and developing an ensemble model for each of the MAQC-II endpoints. We compared the ensemble models with both nominated and candidate models from MAQC-II using blinded external validation. Results For 10 of the 13 MAQC-II endpoints originally analyzed by the MAQC-II data analysis team from the National Center for Toxicological Research (NCTR), the ensemble models achieved equal or better predictive performance than the NCTR nominated models. Additionally, the ensemble models had performance comparable to the MAQC-II candidate models. Most ensemble models also had better performance than the nominated models generated by five other MAQC-II data analysis teams that analyzed all 13 endpoints. Conclusions Our findings suggest that an ensemble method can often attain a higher average predictive performance in an external validation set than a corresponding “optimized” model method. Using an ensemble method to determine a final model is a potentially important supplement to the good modeling practices recommended by the MAQC-II project for developing microarray-based genomic biomarkers.

Gaining Confidence on Molecular Classification Through Consensus Modeling and Validation

Using Decision Forest to Classify Prostate Cancer Samples on the Basis of Seldi-Tof Ms Data: Assessing Chance Correlation and Prediction Confidence

Multiclass Decision Forest - A Novel Pattern Recognition Method For Multiclass Classification In Microarray Data Analysis

Consensus Analysis of Multiple Classifiers Using Non-Repetitive Variables: Diagnostic Application to Microarray Gene Expression Data

A novel classification method of microarray with reliability and confidence

Comprehensive Comparison of Classifiers for Metabolic Profiling Analysis

Assessing concordance among human, in silico predictions and functional assays on genetic variant classification.

Discriminant Models on Mitochondrial Toxicity Improved by Consensus Modeling and Resolving Imbalance in Training.

Random Forest In Clinical Metabolomics For Phenotypic Discrimination And Biomarker Selection

DNA-framework-based multidimensional molecular classifiers for cancer diagnosis

Application of multiple statistical tests to enhance mass spectrometry-based biomarker discovery

On the combination of omics data for prediction of binary outcomes

On the complementarity of the consensus-based disorder prediction.

Validating the validation: reanalyzing a large-scale comparison of deep learning and machine learning models for bioactivity prediction

Stabilizing Machine Learning for Reproducible and Explainable Results: A Novel Validation Approach to Subject-Specific Insights

Shrunken centroids regularized discriminant analysis as a promising strategy for metabolomics data exploration

Critical Assessment of the Biomarker Discovery and Classification Methods for Multiclass Metabolomics.

A Novel SPDF Ensemble Classifier for Cancer Classification

MOSDNET: A multi-omics classification framework using simplified multi-view deep discriminant representation learning and dynamic edge GCN with multi-task learning

Selecting a single model or combining multiple models for microarray-based classifier development? – A comparative analysis based on large and diverse datasets generated from the MAQC-II project

The Robust Classification Model Based on Combinatorial Features