Abstract:Self-supervised learning (SSL) opens up huge opportunities for medical image analysis that is well known for its lack of annotations. However, aggregating massive (unlabeled) 3D medical images like computerized tomography (CT) remains challenging due to its high imaging cost and privacy restrictions. In our pilot study, we advocated bringing a wealth of 2D images like chest X-rays as compensation for the lack of 3D data, aiming to build a universal medical self-supervised representation learning framework, called UniMiSS. Especially, we designed a pyramid U- like medical Transformer (MiT) as the backbone to make UniMiSS possible to perform SSL with both 2D and 3D images. Consequently, the predecessor UniMiSS has two obvious merits compared to current 3D-specific SSL: (1) more effective - superior to learning strong representations, benefiting from more and diverse data; and (2) more versatile - suitable for various downstream tasks without the restriction on the dimensionality barrier. Unfortunately, UniMiSS did not dig deeply into the intrinsic anatomy correlation between 2D medical images and 3D volumes due to the lack of paired multi-modal/dimension patient data. In this extension paper, we propose the UniMiSS+, in which we introduce the digitally reconstructed radiographs (DRR) technology to simulate X-ray images from a CT volume to access paired CT and X-ray data. Benefiting from the paired group, we introduce an extra pair- wise constraint to boost the cross-modality correlation learning, which also can be adopted as a cross-dimension regularization to further improve the representations. We conduct expensive experiments on multiple 3D/2D medical image analysis tasks, including segmentation and classification. The results show that the proposed UniMiSS+ achieves promising performance on various downstream tasks, not only outperforming the ImageNet pre-training and other advanced SSL counterparts substantially but also improving the predecessor UniMiSS pre-training. Code is available at: https://github.com/YtongXie/UniMiSS-code.

Universal Semi-Supervised Learning for Medical Image Classification

Category-Level Regularized Unlabeled-to-Labeled Learning for Semi-supervised Prostate Segmentation with Multi-site Unlabeled Data

Semi-Supervised Convolutional Vision Transformer with Bi-Level Uncertainty Estimation for Medical Image Segmentation

Dive into the Details of Self-Supervised Learning for Medical Image Analysis.

Self-Supervised Learning and Its Applications in Ophthalmology

Twin Self-Supervision Based Semi-Supervised Learning (TS-SSL): Retinal Anomaly Classification in SD-OCT Images

Towards Generic Semi-Supervised Framework for Volumetric Medical Image Segmentation

Deep virtual adversarial self-training with consistency regularization for semi-supervised medical image classification

Dive into Self-Supervised Learning for Medical Image Analysis: Data, Models and Tasks

BoostMIS: Boosting Medical Image Semi-supervised Learning with Adaptive Pseudo Labeling and Informative Active Annotation

Unified 2D and 3D Pre-training for Medical Image Classification and Segmentation.

A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysis

UniMiSS: Universal Medical Self-supervised Learning Via Breaking Dimensionality Barrier

Evidence-based uncertainty-aware semi-supervised medical image segmentation

UniMiSS+: Universal Medical Self-Supervised Learning From Cross-Dimensional Unpaired Data

Fix-A-Step: Semi-supervised Learning from Uncurated Unlabeled Data

Semi-Supervised Medical Image Classification Combined with Unsupervised Deep Clustering

Semi-supervised Learning Regularized by Adversarial Perturbation and Diversity Maximization.

Self-supervised learning for medical image analysis: a comprehensive review

Enhanced Semi-Supervised Medical Image Classification Based on Dynamic Sample Reweighting and Pseudo-Label Guided Contrastive Learning (DSRPGC)

Class-balanced Open-set Semi-supervised Object Detection for Medical Images