Abstract:The interpretation of medical images into a natural language is a developing field of artificial intelligence (AI) called image captioning. This field integrates two branches of artificial intelligence which are computer vision and natural language processing. This is a challenging topic that goes beyond object recognition, segmentation, and classification since it demands an understanding of the relationships between various components in an image and how these objects function as visual representations. The content-based image retrieval (CBIR) uses an image captioning model to generate captions for the user query image. The common architecture of medical image captioning systems consists mainly of an image feature extractor subsystem followed by a caption generation lingual subsystem. We aim in this paper to build an optimized model for histopathological captions of stomach adenocarcinoma endoscopic biopsy specimens. For the image feature extraction subsystem, we did two evaluations; first, we tested 5 different vision models (VGG, ResNet, PVT, SWIN-Large, and ConvNEXT-Large) using (LSTM, RNN, and bidirectional-RNN) and then compare the vision models with (LSTM-without augmentation, LSTM-with augmentation and BioLinkBERT-Large as an embedding layer-with augmentation) to find the accurate one. Second, we tested 3 different concatenations of pairs of vision models (SWIN-Large, PVT_v2_b5, and ConvNEXT-Large) to get among them the most expressive extracted feature vector of the image. For the caption generation lingual subsystem, we tested a pre-trained language embedding model which is BioLinkBERT-Large compared to LSTM in both evaluations, to select from them the most accurate model. Our experiments showed that building a captioning system that uses a concatenation of the two models ConvNEXT-Large and PVT_v2_b5 as an image feature extractor, combined with the BioLinkBERT-Large language embedding model produces the best results among the other combinations.

Enhanced descriptive captioning model for histopathological patches

Towards a Visual-Language Foundation Model for Computational Pathology

Sam-Guided Enhanced Fine-Grained Encoding with Mixed Semantic Learning for Medical Image Captioning

A visual-language foundation model for computational pathology

Bidirectional Captioning for Clinically Accurate and Interpretable Models

Uterine Ultrasound Image Captioning Using Deep Learning Techniques

Automatic Report Generation for Histopathology images using pre-trained Vision Transformers and BERT

DS@BioMed at ImageCLEFmedical Caption 2024: Enhanced Attention Mechanisms in Medical Caption Generation through Concept Detection Integration

A Multimodal Approach For Endoscopic VCE Image Classification Using BiomedCLIP-PubMedBERT

UIT-DarkCow team at ImageCLEFmedical Caption 2024: Diagnostic Captioning for Radiology Images Efficiency with Transformer Models

D-CNN: A New model for Generating Image Captions with Text Extraction Using Deep Learning for Visually Challenged Individuals

Image Captioning using Facial Expression and Attention

What a Whole Slide Image Can Tell? Subtype-guided Masked Transformer for Pathological Image Captioning

Rethinking Surgical Captioning: End-to-End Window-Based MLP Transformer Using Patches

Customizing General-Purpose Foundation Models for Medical Report Generation

PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration

An Intelligent Image Caption Generator Model using Deep Learning

CaptionNet: Automatic End-to-End Siamese Difference Captioning Model with Attention

Image Captioning with Deep Bidirectional LSTMs

Understanding transfer learning for chest radiograph clinical report generation with modified transformer architectures

A Data-Driven Guided Decoding Mechanism for Diagnostic Captioning