Abstract:This study introduces a novel approach to enhance molecular representation learning by integrating high‐content cell microscopy images at the phenotypic level. The proposed unified framework employs contrastive loss functions for cross‐modal pre‐training, enabling mutual retrieval of molecules and images. The model improves not only molecular properties predictions but also clinical outcome predictions, highlighting the potential of cross‐modality learning in bridging molecular structures to phenotypes for drug discovery. Constructing discriminative representations of molecules lies at the core of a number of domains such as drug discovery, chemistry, and medicine. State‐of‐the‐art methods employ graph neural networks and self‐supervised learning (SSL) to learn unlabeled data for structural representations, which can then be fine‐tuned for downstream tasks. Albeit powerful, these methods are pre‐trained solely on molecular structures and thus often struggle with tasks involved in intricate biological processes. Here, it is proposed to assist the learning of molecular representation by using the perturbed high‐content cell microscopy images at the phenotypic level. To incorporate the cross‐modal pre‐training, a unified framework is constructed to align them through multiple types of contrastive loss functions, which is proven effective in the formulated novel tasks to retrieve the molecules and corresponding images mutually. More importantly, the model can infer functional molecules according to cellular images generated by genetic perturbations. In parallel, the proposed model can transfer non‐trivially to molecular property predictions, and has shown great improvement over clinical outcome predictions. These results suggest that such cross‐modality learning can bridge molecules and phenotype to play important roles in drug discovery.

Multi-modal contrastive learning of subcellular organization using DICE

ACE: a versatile contrastive learning framework for single-cell mosaic integration

Multimodal Protein-Ligand Contrastive Pretraining for Effective and Efficient Drug Discovery

CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrieval

Towards robust and generalizable representations of extracellular data using contrastive learning

Cross‐Modal Graph Contrastive Learning with Cellular Images

Comprehensive View Embedding Learning for Single-Cell Multimodal Integration

Progressive Multi-Modality Learning for Inverse Protein Folding

Multimodal Contrastive In-Context Learning

MMCL-CPI: A multi-modal compound-protein interaction prediction model incorporating contrastive learning pre-training

DEDUCE: Multi-head attention decoupled contrastive learning to discover cancer subtypes based on multi-omics data

Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data

Contrastive Representation Learning for 3D Protein Structures

CCPL: Cross-modal Contrastive Protein Learning

Contrastive Learning for Robust Cell Annotation and Representation from Single-Cell Transcriptomics

Multi-marginal Contrastive Learning for Multilabel Subcellular Protein Localization

ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics

scCobra: Contrastive cell embedding learning with domain-adaptation for single-cell data integration and harmonization

Single-cell multi-omics integration for unpaired data by a siamese network with graph-based contrastive loss

Contrastively generative self-expression model for single-cell and spatial multimodal data