A Multimodal Approach For Endoscopic VCE Image Classification Using BiomedCLIP-PubMedBERT

Nagarajan Ganapathy,Podakanti Satyajith Chary,Teja Venkata Ramana Kumar Pithani,Pavan Kavati,Arun Kumar S

2024-10-26

Abstract:This Paper presents an advanced approach for fine-tuning BiomedCLIP PubMedBERT, a multimodal model, to classify abnormalities in Video Capsule Endoscopy (VCE) frames, aiming to enhance diagnostic efficiency in gastrointestinal healthcare. By integrating the PubMedBERT language model with a Vision Transformer (ViT) to process endoscopic images, our method categorizes images into ten specific classes: angioectasia, bleeding, erosion, erythema, foreign body, lymphangiectasia, polyp, ulcer, worms, and normal. Our workflow incorporates image preprocessing and fine-tunes the BiomedCLIP model to generate high-quality embeddings for both visual and textual inputs, aligning them through similarity scoring for classification. Performance metrics, including classification, accuracy, recall, and F1 score, indicate the models strong ability to accurately identify abnormalities in endoscopic frames, showing promise for practical use in clinical diagnostics.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the challenges of abnormal classification in video capsule endoscopy (VCE) images to improve the efficiency of gastrointestinal medical diagnosis. Specifically, the paper focuses on the following aspects: 1. **Challenges of Manual Interpretation**: VCE technology is widely used in gastrointestinal diagnosis, but the thousands of images generated need to be manually interpreted, which is not only time-consuming and prone to errors but also requires professional knowledge and experience. 2. **High Intra-class Variability and Data Imbalance**: Gastrointestinal abnormal images exhibit high intra-class variability and data imbalance, making it difficult for models to generalize to different patient groups and imaging conditions. 3. **Scarcity of Annotated Data**: The annotation of medical imaging data is often limited by privacy issues and the high cost of professional annotation, making it difficult to obtain large-scale annotated data. To address these issues, the paper proposes a multimodal approach by fine-tuning the BiomedCLIP-PubMedBERT model to classify abnormalities in VCE images. This method combines the PubMedBERT language model and Vision Transformer (ViT), capable of handling both image and text inputs, generating high-quality embeddings, and classifying through similarity scoring. The goal of the paper is to demonstrate the effectiveness and accuracy of this approach in automatically detecting abnormalities in VCE images, thereby improving the efficiency and accuracy of clinical diagnosis.

A Multimodal Approach For Endoscopic VCE Image Classification Using BiomedCLIP-PubMedBERT

Capsule Endoscopy Multi-classification via Gated Attention and Wavelet Transformations

Multi-Class Abnormality Classification Task in Video Capsule Endoscopy

Integrating Deep Feature Extraction and Hybrid ResNet-DenseNet Model for Multi-Class Abnormality Detection in Endoscopic Images

Video Capsule Endoscopy Classification using Focal Modulation Guided Convolutional Neural Network

ViTCA-Net: a framework for disease detection in video capsule endoscopy images using a vision transformer and convolutional neural network with a specific attention mechanism

Efficacy of a comprehensive binary classification model using a deep convolutional neural network for wireless capsule endoscopy

The intersection of video capsule endoscopy and artificial intelligence: addressing unique challenges using machine learning

Exploring vision transformers for classifying early Barrett's dysplasia in endoscopic images: A pilot study on white-light and narrow-band imaging

Classification of Endoscopy and Video Capsule Images using CNN-Transformer Model

P469 Deep learning and minimally invasive endoscopy – panendoscopic detection of pleomorphic lesions

Domain-Adaptive Pre-training of Self-Supervised Foundation Models for Medical Image Classification in Gastrointestinal Endoscopy

Multi-view Multi-modal Feature Embedding for Endomicroscopy Mosaic Classification

PS-DeVCEM: Pathology-sensitive deep learning model for video capsule endoscopy based on weakly labeled data

Celiac Disease Diagnosis from Videocapsule Endoscopy Images with Residual Learning and Deep Feature Extraction.

BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs

Reduction of Video Capsule Endoscopy Reading Times Using Deep Learning with Small Data

ClassifyViStA:WCE Classification with Visual understanding through Segmentation and Attention

A comprehensive analysis of classification methods in gastrointestinal endoscopy imaging

Language Augmentation in CLIP for Improved Anatomy Detection on Multi-modal Medical Images

Multi-Class Abnormality Classification in Video Capsule Endoscopy Using Deep Learning