Abstract:Self-supervised learning (SSL) opens up huge opportunities for medical image analysis that is well known for its lack of annotations. However, aggregating massive (unlabeled) 3D medical images like computerized tomography (CT) remains challenging due to its high imaging cost and privacy restrictions. In our pilot study, we advocated bringing a wealth of 2D images like chest X-rays as compensation for the lack of 3D data, aiming to build a universal medical self-supervised representation learning framework, called UniMiSS. Especially, we designed a pyramid U- like medical Transformer (MiT) as the backbone to make UniMiSS possible to perform SSL with both 2D and 3D images. Consequently, the predecessor UniMiSS has two obvious merits compared to current 3D-specific SSL: (1) more effective - superior to learning strong representations, benefiting from more and diverse data; and (2) more versatile - suitable for various downstream tasks without the restriction on the dimensionality barrier. Unfortunately, UniMiSS did not dig deeply into the intrinsic anatomy correlation between 2D medical images and 3D volumes due to the lack of paired multi-modal/dimension patient data. In this extension paper, we propose the UniMiSS+, in which we introduce the digitally reconstructed radiographs (DRR) technology to simulate X-ray images from a CT volume to access paired CT and X-ray data. Benefiting from the paired group, we introduce an extra pair- wise constraint to boost the cross-modality correlation learning, which also can be adopted as a cross-dimension regularization to further improve the representations. We conduct expensive experiments on multiple 3D/2D medical image analysis tasks, including segmentation and classification. The results show that the proposed UniMiSS+ achieves promising performance on various downstream tasks, not only outperforming the ImageNet pre-training and other advanced SSL counterparts substantially but also improving the predecessor UniMiSS pre-training. Code is available at: https://github.com/YtongXie/UniMiSS-code.

A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysis

PCRLv2: A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysis

MsVRL: Self-Supervised Multiscale Visual Representation Learning Via Cross-Level Consistency for Medical Image Segmentation

Unified 2D and 3D Pre-training for Medical Image Classification and Segmentation.

Dive into the Details of Self-Supervised Learning for Medical Image Analysis.

Unified Feature Consistency of Under-Performing Pixels and Valid Regions for Semi-Supervised Medical Image Segmentation

Uni-Dual: A Generic Unified Dual-Task Medical Self-Supervised Learning Framework

Uni4Eye++: A General Masked Image Modeling Multi-modal Pre-training Framework for Ophthalmic Image Classification and Segmentation

UniMiSS+: Universal Medical Self-Supervised Learning From Cross-Dimensional Unpaired Data

UniMiSS: Universal Medical Self-supervised Learning Via Breaking Dimensionality Barrier

Self-Supervised Learning Featuring Small-Scale Image Dataset for Treatable Retinal Diseases Classification

Dive into Self-Supervised Learning for Medical Image Analysis: Data, Models and Tasks

Keypoint-Augmented Self-Supervised Learning for Medical Image Segmentation with Limited Annotation

Unified Medical Image Pre-training in Language-Guided Common Semantic Space

Self-supervised dual-head attentional bootstrap learning network for prostate cancer screening in transrectal ultrasound images

Uni4Eye: Unified 2D and 3D Self-supervised Pre-training via Masked Image Modeling Transformer for Ophthalmic Image Classification

Self-Supervised Pre-Training Based on Contrastive Complementary Masking for Semi-Supervised Cardiac Image Segmentation

Enhanced Self-supervised Learning for Multi-modality MRI Segmentation and Classification: A Novel Approach Avoiding Model Collapse

SC-SSL: Self-correcting Collaborative and Contrastive Co-training Model for Semi-Supervised Medical Image Segmentation

A knowledge-based learning framework for self-supervised pre-training towards enhanced recognition of biomedical microscopy images

UniVIP: A Unified Framework for Self-Supervised Visual Pre-training