Self-supervised learning of multi-omics embeddings in the low-label, high-data regime

Christian John Hurry,Emma Slade

2023-11-16

Abstract:Contrastive, self-supervised learning (SSL) is used to train a model that predicts cancer type from miRNA, mRNA or RPPA expression data. This model, a pretrained FT-Transformer, is shown to outperform XGBoost and CatBoost, standard benchmarks for tabular data, when labelled samples are scarce but the number of unlabelled samples is high. This is despite the fact that the datasets we use have $\mathcal{O}(10^{1})$ classes and $\mathcal{O}(10^{2})-\mathcal{O}(10^{4})$ features. After demonstrating the efficacy of our chosen method of self-supervised pretraining, we investigate SSL for multi-modal models. A late-fusion model is proposed, where each omics is passed through its own sub-network, the outputs of which are averaged and passed to the pretraining or downstream objective function. Multi-modal pretraining is shown to improve predictions from a single omics, and we argue that this is useful for datasets with many unlabelled multi-modal samples, but few labelled unimodal samples. Additionally, we show that pretraining each omics-specific module individually is highly effective. This enables the application of the proposed model in a variety of contexts where a large amount of unlabelled data is available from each omics, but only a few labelled samples.

Machine Learning,Quantitative Methods

What problem does this paper attempt to address?

The paper primarily aims to address the following two core issues:

1. **Developing effective self-supervised learning methods to handle multi-omics data**: Given the scenario in biomedical research where labeled samples are scarce but unlabeled samples are abundant, the researchers propose a self-supervised pre-training method based on contrastive learning (Mask Token Replacement, MTR) for representation learning of multi-omics data. This method leverages a large amount of unlabeled multi-omics data to learn useful feature representations and performs well even when labeled samples are very limited.

2. **Construction and optimization of multimodal models**: The researchers also explore how to effectively integrate data from different omics technologies (such as miRNA, mRNA, RPPA, etc.) by constructing a novel multimodal fusion model. This model allows each omics data to be processed through its respective sub-network, and then the outputs of these sub-networks are averaged and passed to the objective function of the pre-training or downstream tasks. This approach not only improves the accuracy of single-omics predictions but also effectively handles the situation of missing modalities in the data.

In short, this paper aims to improve the analysis methods of multi-omics data through self-supervised learning strategies, particularly enhancing model performance in scenarios where labeled data is scarce but unlabeled data is abundant. Additionally, it proposes a flexible multimodal model architecture that better integrates various types of omics data, thereby improving the prediction capability for complex diseases.