Abstract:Self-supervised pre-training has become the priory choice to establish reliable neural networks for automated recognition of massive biomedical microscopy images, which are routinely annotation-free, without semantics, and without guarantee of quality. Note that this paradigm is still at its infancy and limited by closely related open issues: (1) how to learn robust representations in an unsupervised manner from unlabelled biomedical microscopy images of low diversity in samples? and (2) how to obtain the most significant representations demanded by a high-quality segmentation? Aiming at these issues, this study proposes a knowledge-based learning framework (TOWER) towards enhanced recognition of biomedical microscopy images, which works in three phases by synergizing contrastive learning and generative learning methods: (1) Sample Space Diversification: Reconstructive proxy tasks have been enabled to embed a priori knowledge with context highlighted to diversify the expanded sample space; (2) Enhanced Representation Learning: Informative noise-contrastive estimation loss regularizes the encoder to enhance representation learning of annotation-free images; (3) Correlated Optimization: Optimization operations in pre-training the encoder and the decoder have been correlated via image restoration from proxy tasks, targeting the need for semantic segmentation. Experiments have been conducted on public datasets of biomedical microscopy images against the state-of-the-art counterparts (e.g., SimCLR and BYOL), and results demonstrate that: TOWER statistically excels in all self-supervised methods, achieving a Dice improvement of 1.38 percentage points over SimCLR. TOWER also has potential in multi-modality medical image analysis and enables label-efficient semi-supervised learning, e.g., reducing the annotation cost by up to 99% in pathological classification.

MIR: A Benchmark for Molecular Image Retrival with a Cross-modal Pretraining Framework.

An Image-enhanced Molecular Graph Representation Learning Framework

Multistage and Multi-features Medical Image Retrieval System

SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval

3D-MIR: A Benchmark and Empirical Study on 3D Medical Image Retrieval in Radiology

Chemical structure-aware molecular image representation learning

MITER: Medical Image–TExt joint adaptive pretRaining with multi-level contrastive learning

Region-based Contrastive Pretraining for Medical Image Retrieval with Anatomic Query

ProBio: A Protocol-guided Multimodal Dataset for Molecular Biology Lab

MaskMol: Knowledge-guided Molecular Image Pre-Training Framework for Activity Cliffs

MoleMCL: a multi-level contrastive learning framework for molecular pre-training

Dual-view Molecular Pre-training

BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations

Universal Model for Multi-Domain Medical Image Retrieval

A knowledge-based learning framework for self-supervised pre-training towards enhanced recognition of biomedical microscopy images

SMiCRM: A Benchmark Dataset of Mechanistic Molecular Images

Towards Cross-Modal Text-Molecule Retrieval with Better Modality Alignment

AutoMIR: Effective Zero-Shot Medical Information Retrieval without Relevance Labels

BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs

Exploring Optimal Transport-Based Multi-Grained Alignments for Text-Molecule Retrieval

PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents