Abstract:In medical imaging, the classification and segmentation of lesions have always been significant topics in clinical research. Different categories of lesions require different treatment strategies, and accurate segmentation helps to assist in improving the effect of the clinical treatment. The Segment anything model (SAM) is an image segmentation model trained on a large-scale dataset with strong prompt segmentation capability, but it cannot be directly applied to the classification and segmentation tasks of medical images due to insufficient training on medical image data. In this paper, we propose a deep learning method for the classification and segmentation of lesions, called GMM-based segment anything model (G-SAM). Prompt-tuning is utilized in the model with the LoRA strategy, and the lesion feature extraction (GFE) module based on the Gaussian mixture model (GMM), is designed to effectively improve the effect of lesion classification and segmentation on the basis of the SAM. Notably, G-SAM exhibits greater sensitivity to early stage of the lesions, aiding in tumor detection and prevention, which holds important clinical value. G-SAM overcomes the limitation that SAM is not suitable for the medical image classification and segmentation tasks due to insufficient training data with minimal cost. Moreover, it enhances classification accuracy and segmentation precision compared to traditional Gaussian model-based methods. The effectiveness of G-SAM in classifying and segmenting lesions is validated on the LIDC dataset, demonstrating advantages over state-of-the-art (SOTA) methods. The study further validates the applicability of G-SAM on large publicly available datasets across three different image modalities, achieving superior performance.

Sam-Guided Enhanced Fine-Grained Encoding with Mixed Semantic Learning for Medical Image Captioning

$\mathrm{SAM^{Med}}$: A medical image annotation framework based on large vision model

CC-SAM: SAM with Cross-feature Attention and Context for Ultrasound Image Segmentation

SAM-IE: SAM-based Image Enhancement for Facilitating Medical Image Diagnosis with Segmentation Foundation Model

Boosting Medical Image Classification with Segmentation Foundation Model

SAM-Med2D

G-SAM: GMM-based segment anything model for medical image classification and segmentation

SamDSK: Combining Segment Anything Model with Domain-Specific Knowledge for Semi-Supervised Learning in Medical Image Segmentation

MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation

TG-LMM: Enhancing Medical Image Segmentation Accuracy through Text-Guided Large Multi-Modal Model

AGSAM: Agent-Guided Segment Anything Model for Automatic Segmentation in Few-Shot Scenarios

Segment and Caption Anything

How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model

Increasing SAM Zero-Shot Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotation

SemiSAM: Enhancing Semi-Supervised Medical Image Segmentation via SAM-Assisted Consistency Regularization

Medical image analysis using improved SAM-Med2D: segmentation and classification perspectives

Improving Segment Anything on the Fly: Auxiliary Online Learning and Adaptive Fusion for Medical Image Segmentation

Segment anything model for medical image segmentation: Current applications and future directions

Towards Segment Anything Model (SAM) for Medical Image Segmentation: A Survey

MedCLIP-SAM: Bridging Text and Image Towards Universal Medical Image Segmentation

A Survey of Medical Image Captioning Technique: Encoding, Decoding and Latest Advance