Impact of Molecular Representations on Deep Learning Model Comparisons in Drug Response Predictions

Gihan Panapitiya,Rajeev Jain,Carter Knuston,Andrew McNaughton, Justin M. Wozniak, Thomas Brettin, Neeraj Kumar, Rick Stevens
DOI: https://doi.org/10.26434/chemrxiv-2024-g95j5-v2
2024-09-02
Abstract:Deep learning (DL) plays a crucial role in tackling the complexity and heterogeneity of cancer, particularly in predicting drug response. However, the effectiveness of these models is often hindered by inconsistent benchmarks and disparate data sources. To address the gaps in comparisons, we introduce CoMParison workflow for Cross Validation (CMP-CV), an automated cross-validation framework that trains multiple models with user-specified parameters and evaluation metrics. The effectiveness of DL models in predicting drug responses is closely tied to the methods used to represent drugs at the molecular level. In this contribution, we benchmarked commonly leveraged drug representations (graph, molecular descriptors, molecular fingerprints, and SMILES) to learn and understand the predictive capabilities of the models. We compare the ability of different drug representations to encode different structural properties of the drugs by using prediction errors made by models in different drug descriptor domains. We find that, in terms of the average prediction error over the entire test set, molecular descriptors and Morgan fingerprints perform slightly better than the others. However, we also observe that the rankings of the model performance vary in different regions over the descriptor space studied in this work, emphasizing the importance of domain-based model comparison when selecting a model for a specific application. Our efforts are part of CANcer Distributed Learning Environment (CANDLE), enhancing the model comparison capabilities in cancer research and driving the development of more effective strategies for drug response prediction and optimization.
Chemistry
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve The paper aims to address the issue of evaluating the performance of Drug Response Prediction (DRP) models in cancer research. Specifically: 1. **Inconsistency in Model Comparison**: In current research, different research teams often develop custom model implementations and validation strategies, making it difficult to consistently evaluate model capabilities across different drug representation methods, model architectures, and datasets. 2. **Lack of Standardized Comparison Methods**: Traditional model performance evaluation methods rely on scores from original publications, leading to incomparable and inconsistent results, hindering the understanding of key factors affecting predictive performance. To address these issues, the authors propose an automated cross-validation framework called CoMParison workflow for Cross Validation (CMP-CV). CMP-CV allows for the simultaneous training and evaluation of multiple deep learning models and provides infrastructure support through standardized datasets, preprocessing methods, and performance metrics to enable controlled experiments. Additionally, CMP-CV supports user-defined Python functions for deeper analysis of model representations and uncertainties. Using the CMP-CV framework, the authors compared the performance of different drug representation methods (such as graph structures, molecular descriptors, Morgan fingerprints, and SMILES encoding) in drug response prediction and found that certain models perform better in specific drug descriptor domains. This domain-based model comparison helps in selecting the best model suited for specific applications. Furthermore, the authors developed a web application that allows users to find the most suitable model for specific drug candidates based on various molecular descriptors. In summary, the paper aims to enhance the comparability of drug response prediction models through a standardized and comprehensive approach, thereby advancing more effective drug response prediction and optimization strategies in cancer research.