Abstract:Most advances in medical image recognition supporting clinical auxiliary diagnosis meet challenges due to the low-resource situation in the medical field, where annotations are highly expensive and professional. This low-resource problem can be alleviated by leveraging the transferable representations of large-scale pre-trained vision-language models like CLIP. After being pre-trained using large-scale unlabeled medical images and texts (such as medical reports), the vision-language models can learn transferable representations and support flexible downstream clinical tasks such as medical image classification via relevant medical text prompts. However, existing pre-trained vision-language models require domain experts (clinicians) to carefully design the medical text prompts based on different datasets when applied to specific medical image tasks, which is extremely time-consuming and greatly increases the burden on clinicians. To address this problem, we propose a weakly supervised prompt learning method MedPrompt for automatically generating medical prompts, which includes an unsupervised pre-trained vision-language model and a weakly supervised prompt learning model. The unsupervised pre-trained vision-language model adopts large-scale medical images and texts for pre-training, utilizing the natural correlation between medical images and corresponding medical texts without manual annotations. The weakly supervised prompt learning model only utilizes the classes of images in the dataset to guide the learning of the specific class vector in the prompt, while the learning of other context vectors in the prompt does not require any manual annotations for guidance. To the best of our knowledge, this is the first model to automatically generate medical prompts. With the assistance of these prompts, the pre-trained vision-language model can be freed from the strong expert dependency of manual annotation and manual prompt design, thus achieving end-to-end, low-cost medical image classification. Experimental results show that the model using our automatically generated prompts outperforms all its hand-crafted prompts counterparts in full-shot learning on all four datasets, and achieves superior accuracy on zero-shot image classification and few-shot learning in three of the four medical benchmark datasets and comparable accuracy in the remaining one. In addition, the proposed prompt generator is lightweight and therefore has the potential to be embedded into any network architecture.

Few-Shot and Prompt Training for Text Classification in German Doctor's Letters

Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Prompting

Clinical information extraction for lower-resource languages and domains with few-shot learning using pretrained language models and prompting

Zero-shot prompt-based classification: topic labeling in times of foundation models in German Tweets

Automatic extraction of 12 cardiovascular concepts from German discharge letters using pre-trained language models

Keyword-optimized Template Insertion for Clinical Information Extraction via Prompt-based Learning

Medical text classification based on the discriminative pre-training model and prompt-tuning

Comprehensive Study on German Language Models for Clinical and Biomedical Text Understanding

Zero-Shot LLMs for Named Entity Recognition: Targeting Cardiac Function Indicators in German Clinical Texts

HealthPrompt: A Zero-shot Learning Paradigm for Clinical Natural Language Processing

Pathology-knowledge Enhanced Multi-instance Prompt Learning for Few-shot Whole Slide Image Classification

Clinical Prompt Learning With Frozen Language Models

Exploring low-resource medical image classification with weakly supervised prompt learning

PromptBoosting: Black-Box Text Classification with Ten Forward Passes

Automated HEART score determination via ChatGPT: Honing a framework for iterative prompt development

Prompt-Learning for Short Text Classification

A pen mark is all you need - Incidental prompt injection attacks on Vision Language Models in real-life histopathology

Deep Learning Approaches Outperform Conventional Strategies in De-Identification of German Medical Reports

A Keyword-Enhanced Approach to Handle Class Imbalance in Clinical Text Classification

Few-Shot Learning for Chronic Disease Management: Leveraging Large Language Models and Multi-Prompt Engineering with Medical Knowledge Injection

The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification