Abstract:Few-shot learning has been successfully applied to medical image classification as only very few medical examples are available for training. Due to the challenging problem of limited number of annotated medical images, image representations should not be solely derived from a single image modality which is insufficient for characterizing concept classes. In this paper, we propose a new prompting multi-modal model paradigm on medical image classification based on multi-modal foundation models, called PM2. Besides image modality,PM2 introduces another supplementary text input, known as prompt, to further describe corresponding image or concept classes and facilitate few-shot learning across diverse modalities. To better explore the potential of prompt engineering, we empirically investigate five distinct prompt schemes under the new paradigm. Furthermore, linear probing in multi-modal models acts as a linear classification head taking as input only class token, which ignores completely merits of rich statistics inherent in high-level visual tokens. Thus, we alternatively perform a linear classification on feature distribution of visual tokens and class token simultaneously. To effectively mine such rich statistics, a global covariance pooling with efficient matrix power normalization is used to aggregate visual tokens. Then we study and combine two classification heads. One is shared for class token of image from vision encoder and prompt representation encoded by text encoder. The other is to classification on feature distribution of visual tokens from vision encoder. Extensive experiments on three medical datasets show that our PM2 significantly outperforms counterparts regardless of prompt schemes and achieves state-of-the-art performance.

Modal-aware Visual Prompting for Incomplete Multi-modal Brain Tumor Segmentation

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

Unveiling Incomplete Modality Brain Tumor Segmentation: Leveraging Masked Predicted Auto-Encoder and Divergence Learning

Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion

Medical Visual Prompting (MVP): A Unified Framework for Versatile and High-Quality Medical Image Segmentation

BrainMVP: Multi-modal Vision Pre-training for Brain Image Analysis using Multi-parametric MRI

Query Re-Training for Modality-Gnostic Incomplete Multi-modal Brain Tumor Segmentation.

Ma2SP: Missing-Aware Prompting With Modality-Adaptive Integration for Incomplete Multi-Modal Survival Prediction

MedMAP: Promoting Incomplete Multi-modal Brain Tumor Segmentation with Alignment

Modality-Aware and Shift Mixer for Multi-modal Brain Tumor Segmentation

Deep Correlated Prompting for Visual Recognition with Missing Modalities

Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

Modality-Specific Information Disentanglement from Multi-parametric MRI for Breast Tumor Segmentation and Computer-aided Diagnosis

DIGEST: Deeply supervIsed knowledGE tranSfer neTwork learning for brain tumor segmentation with incomplete multi-modal MRI scans

Modality-Pairing Learning for Brain Tumor Segmentation

Pre-Post Interaction Learning for Brain Tumor Segmentation with Missing MRI Modalities.

PM2: A New Prompting Multi-modal Model Paradigm for Few-shot Medical Image Classification

Brain Tumor Segmentation on MRI with Missing Modalities

CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation

Deformation-aware and reconstruction-driven multimodal representation learning for brain tumor segmentation with missing modalities