A Clinical Neurophysiology Information System based on Tcl/Tk

M. Andrews,R. Burgess

1996-07-10

Abstract:The Department of Neurology at the Cleveland Clinic has replaced a restricted patient information system with a new one based on Tcl/Tk. Tcl/Tk along with several novel extensions provides the base for creating portable client-server database applications. These extensions and some of the clinical applications are presented.

What problem does this paper attempt to address?

Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning

Weijian Huang,Cheng Li,Hong-Yu Zhou,Hao Yang,Jiarun Liu,Yong Liang,Hairong Zheng,Shaoting Zhang,Shanshan Wang

DOI: https://doi.org/10.1038/s41467-024-51749-0

2024-09-03

Abstract:Recently, multi-modal vision-language foundation models have gained significant attention in the medical field. While these models offer great opportunities, they still face crucial challenges, such as the requirement for fine-grained knowledge understanding in computer-aided diagnosis and the capability of utilizing very limited or even no task-specific labeled data in real-world clinical applications. In this study, we present MaCo, a masked contrastive chest X-ray foundation model that tackles these challenges. MaCo explores masked contrastive learning to simultaneously achieve fine-grained image understanding and zero-shot learning for a variety of medical imaging tasks. It designs a correlation weighting mechanism to adjust the correlation between masked chest X-ray image patches and their corresponding reports, thereby enhancing the model's representation learning capabilities. To evaluate the performance of MaCo, we conducted extensive experiments using 6 well-known open-source X-ray datasets. The experimental results demonstrate the superiority of MaCo over 10 state-of-the-art approaches across tasks such as classification, segmentation, detection, and phrase grounding. These findings highlight the significant potential of MaCo in advancing a wide range of medical image analysis tasks.

Computer Vision and Pattern Recognition
MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models

Hari Sowrirajan,Jingbo Yang,Andrew Y. Ng,Pranav Rajpurkar

DOI: https://doi.org/10.48550/arXiv.2010.05352

2021-05-18

Abstract:Contrastive learning is a form of self-supervision that can leverage unlabeled data to produce pretrained models. While contrastive learning has demonstrated promising results on natural image classification tasks, its application to medical imaging tasks like chest X-ray interpretation has been limited. In this work, we propose MoCo-CXR, which is an adaptation of the contrastive learning method Momentum Contrast (MoCo), to produce models with better representations and initializations for the detection of pathologies in chest X-rays. In detecting pleural effusion, we find that linear models trained on MoCo-CXR-pretrained representations outperform those without MoCo-CXR-pretrained representations, indicating that MoCo-CXR-pretrained representations are of higher-quality. End-to-end fine-tuning experiments reveal that a model initialized via MoCo-CXR-pretraining outperforms its non-MoCo-CXR-pretrained counterpart. We find that MoCo-CXR-pretraining provides the most benefit with limited labeled training data. Finally, we demonstrate similar results on a target Tuberculosis dataset unseen during pretraining, indicating that MoCo-CXR-pretraining endows models with representations and transferability that can be applied across chest X-ray datasets and tasks.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning
MCL: Multi-view Enhanced Contrastive Learning for Chest X-ray Report Generation

Kang Liu,Zhuoqi Ma,Kun Xie,Zhicheng Jiao,Qiguang Miao

2024-11-15

Abstract:Radiology reports are crucial for planning treatment strategies and enhancing doctor-patient communication, yet manually writing these reports is burdensome for radiologists. While automatic report generation offers a solution, existing methods often rely on single-view radiographs, limiting diagnostic accuracy. To address this problem, we propose MCL, a Multi-view enhanced Contrastive Learning method for chest X-ray report generation. Specifically, we first introduce multi-view enhanced contrastive learning for visual representation by maximizing agreements between multi-view radiographs and their corresponding report. Subsequently, to fully exploit patient-specific indications (e.g., patient's symptoms) for report generation, we add a transitional ``bridge" for missing indications to reduce embedding space discrepancies caused by their presence or absence. Additionally, we construct Multi-view CXR and Two-view CXR datasets from public sources to support research on multi-view report generation. Our proposed MCL surpasses recent state-of-the-art methods across multiple datasets, achieving a 5.0% F1 RadGraph improvement on MIMIC-CXR, a 7.3% BLEU-1 improvement on MIMIC-ABN, a 3.1% BLEU-4 improvement on Multi-view CXR, and an 8.2% F1 CheXbert improvement on Two-view CXR.

Computer Vision and Pattern Recognition,Artificial Intelligence
A Comparison of Maternal Interview and Medical Record Ascertainment of Violence among Women who had Poor Pregnancy Outcomes

L. Fried,A. Aschengrau,H. Cabral,H. Amaro

DOI: https://doi.org/10.1007/s10995-006-0108-9

2006-06-27

Maternal and Child Health Journal

Abstract:
MKCL: Medical Knowledge with Contrastive Learning model for radiology report generation

Xiaodi Hou,Zhi Liu,Xiaobo Li,Xingwang Li,Shengtian Sang,Yijia Zhang

DOI: https://doi.org/10.1016/j.jbi.2023.104496

IF: 8

2023-10-01

Journal of Biomedical Informatics

Abstract:Automatic radiology report generation has the potential to alert inexperienced radiologists to misdiagnoses or missed diagnoses and improve healthcare delivery efficiency by reducing the documentation workload of radiologists. Motivated by the continuous development of automatic image captioning, more and more deep learning methods have been proposed for automatic radiology report generation. However, the visual and textual data bias problem still face many challenges in the medical domain. Additionally, do not integrate medical knowledge, ignoring the mutual influences between medical findings, and abundant unlabeled medical images influence the accuracy of generating report. In this paper, we propose a Medical Knowledge with Contrastive Learning model (MKCL) to enhance radiology report generation. The proposed model MKCL uses IU Medical Knowledge Graph (IU-MKG) to mine the relationship among medical findings and improve the accuracy of identifying positive diseases findings from radiologic medical images. In particular, we design Knowledge Enhanced Attention (KEA), which integrates the IU-MKG and the extracted chest radiological visual features to alleviate textual data bias. Meanwhile, this paper leverages supervised contrastive learning to relieve radiographic medical images which have not been labeled, and identify abnormalities from images. Experimental results on the public dataset IU X-ray show that our proposed model MKCL outperforms other state-of-the-art report generation methods. Ablation studies also demonstrate that IU medical knowledge graph module and supervised contrastive learning module enhance the ability of the model to detect the abnormal parts and accurately describe the abnormal findings. The source code is available at: https://github.com/Eleanorhxd/MKCL.

medical informatics,computer science, interdisciplinary applications
Radiology Reports Improve Visual Representations Learned from Radiographs

Haoxu Huang,Samyak Rawlekar,Sumit Chopra,Cem M Deniz

Abstract:Although human's ability to visually understand the structure of the World plays a crucial role in perceiving the World and making appropriate decisions, human perception does not solely rely on vision but amalgamates the information from acoustic, verbal, and visual stimuli. An active area of research has been revolving around designing an efficient framework that adapts to multiple modalities and ideally improves the performance of existing tasks. While numerous frameworks have proved effective on natural datasets like ImageNet, a limited number of studies have been carried out in the biomedical domain. In this work, we extend the available frameworks for natural data to biomedical data by leveraging the abundant, unstructured multi-modal data available as radiology images and reports. We attempt to answer the question, "For multi-modal learning, self-supervised learning and joint learning using both learning strategies, which one improves the visual representation for downstream chest radiographs classification tasks the most?". Our experiments indicated that in limited labeled data settings with 1% and 10% labeled data, the joint learning with multi-modal and self-supervised models outperforms self-supervised learning and is at par with multi-modal learning. Additionally, we found that multi-modal learning is generally more robust on out-of-distribution datasets. The code is publicly available online.
Masked Contrastive Reconstruction for Cross-modal Medical Image-Report Retrieval

Zeqiang Wei,Kai Jin,Xiuzhuang Zhou

DOI: https://doi.org/10.48550/arXiv.2312.15840

2023-12-27

Abstract:Cross-modal medical image-report retrieval task plays a significant role in clinical diagnosis and various medical generative tasks. Eliminating heterogeneity between different modalities to enhance semantic consistency is the key challenge of this task. The current Vision-Language Pretraining (VLP) models, with cross-modal contrastive learning and masked reconstruction as joint training tasks, can effectively enhance the performance of cross-modal retrieval. This framework typically employs dual-stream inputs, using unmasked data for cross-modal contrastive learning and masked data for reconstruction. However, due to task competition and information interference caused by significant differences between the inputs of the two proxy tasks, the effectiveness of representation learning for intra-modal and cross-modal features is limited. In this paper, we propose an efficient VLP framework named Masked Contrastive and Reconstruction (MCR), which takes masked data as the sole input for both tasks. This enhances task connections, reducing information interference and competition between them, while also substantially decreasing the required GPU memory and training time. Moreover, we introduce a new modality alignment strategy named Mapping before Aggregation (MbA). Unlike previous methods, MbA maps different modalities to a common feature space before conducting local feature aggregation, thereby reducing the loss of fine-grained semantic information necessary for improved modality alignment. Qualitative and quantitative experiments conducted on the MIMIC-CXR dataset validate the effectiveness of our approach, demonstrating state-of-the-art performance in medical cross-modal retrieval tasks.

Computer Vision and Pattern Recognition,Artificial Intelligence
MoRE: Multi-Modal Contrastive Pre-training with Transformers on X-Rays, ECGs, and Diagnostic Report

Samrajya Thapa,Koushik Howlader,Subhankar Bhattacharjee,Wei le

2024-10-23

Abstract:In this paper, we introduce a novel Multi-Modal Contrastive Pre-training Framework that synergistically combines X-rays, electrocardiograms (ECGs), and radiology/cardiology reports. Our approach leverages transformers to encode these diverse modalities into a unified representation space, aiming to enhance diagnostic accuracy and facilitate comprehensive patient assessments. We utilize LoRA-Peft to significantly reduce trainable parameters in the LLM and incorporate recent linear attention dropping strategy in the Vision Transformer(ViT) for smoother attention. Furthermore, we provide novel multimodal attention explanations and retrieval for our model. To the best of our knowledge, we are the first to propose an integrated model that combines X-ray, ECG, and Radiology/Cardiology Report with this approach. By utilizing contrastive loss, MoRE effectively aligns modality-specific features into a coherent embedding, which supports various downstream tasks such as zero-shot classification and multimodal retrieval. Employing our proposed methodology, we achieve state-of-the-art (SOTA) on the Mimic-IV, CheXpert, Edema Severity, and PtbXl downstream datasets, surpassing existing multimodal approaches. Our proposed framework shows significant improvements in capturing intricate inter-modal relationships and its robustness in medical diagnosis that establishes a framework for future research in multimodal learning in the healthcare sector.

Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning
Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques

Weijian Huang,Cheng Li,Hong-Yu Zhou,Jiarun Liu,Hao Yang,Yong Liang,Guangming Shi,Hairong Zheng,Shanshan Wang

2024-02-26

Abstract:The development of medical vision-language foundation models has attracted significant attention in the field of medicine and healthcare due to their promising prospect in various clinical applications. While previous studies have commonly focused on feature learning at a single learning scale, investigation on integrating multi-scale information is lacking, which may hinder the potential for mutual reinforcement among these features. This paper aims to bridge this gap by proposing a method that effectively exploits multi-scale information to enhance the performance of medical foundation models. The proposed method simultaneously exploits features at the local, instance, modality and global aspects, facilitating comprehensive representation learning within the models. We evaluate the effectiveness of the proposed method on six open-source datasets across different clinical tasks, demonstrating its ability to enhance the performance of medical foundation models.

Computer Vision and Pattern Recognition
Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation

Xi Zhang,Zaiqiao Meng,Jake Lever,Edmond S. L. Ho

DOI: https://doi.org/10.18653/v1/2024.bionlp-1.54

2024-12-06

Abstract:We introduce a radiology-focused visual language model designed to generate radiology reports from chest X-rays. Building on previous findings that large language models (LLMs) can acquire multimodal capabilities when aligned with pretrained vision encoders, we demonstrate similar potential with chest X-ray images. This integration enhances the ability of model to understand and describe chest X-ray images. Our model combines an image encoder with a fine-tuned LLM based on the Vicuna-7B architecture, enabling it to generate different sections of a radiology report with notable accuracy. The training process involves a two-stage approach: (i) initial alignment of chest X-ray features with the LLM (ii) followed by fine-tuning for radiology report generation.

Computer Vision and Pattern Recognition,Computation and Language,Machine Learning
From Few to More: Scribble-based Medical Image Segmentation via Masked Context Modeling and Continuous Pseudo Labels

Zhisong Wang,Yiwen Ye,Ziyang Chen,Minglei Shu,Yong Xia

2024-08-23

Abstract:Scribble-based weakly supervised segmentation techniques offer comparable performance to fully supervised methods while significantly reducing annotation costs, making them an appealing alternative. Existing methods often rely on auxiliary tasks to enforce semantic consistency and use hard pseudo labels for supervision. However, these methods often overlook the unique requirements of models trained with sparse annotations. Since the model must predict pixel-wise segmentation maps with limited annotations, the ability to handle varying levels of annotation richness is critical. In this paper, we adopt the principle of `from few to more' and propose MaCo, a weakly supervised framework designed for medical image segmentation. MaCo employs masked context modeling (MCM) and continuous pseudo labels (CPL). MCM uses an attention-based masking strategy to disrupt the input image, compelling the model's predictions to remain consistent with those of the original image. CPL converts scribble annotations into continuous pixel-wise labels by applying an exponential decay function to distance maps, resulting in continuous maps that represent the confidence of each pixel belonging to a specific category, rather than using hard pseudo labels. We evaluate MaCo against other weakly supervised methods using three public datasets. The results indicate that MaCo outperforms competing methods across all datasets, setting a new record in weakly supervised medical image segmentation.

Computer Vision and Pattern Recognition
Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning

Fuying Wang,Yuyin Zhou,Shujun Wang,Varut Vardhanabhuti,Lequan Yu

DOI: https://doi.org/10.48550/arXiv.2210.06044

2022-10-12

Abstract:Learning medical visual representations directly from paired radiology reports has become an emerging topic in representation learning. However, existing medical image-text joint learning methods are limited by instance or local supervision analysis, ignoring disease-level semantic correspondences. In this paper, we present a novel Multi-Granularity Cross-modal Alignment (MGCA) framework for generalized medical visual representation learning by harnessing the naturally exhibited semantic correspondences between medical image and radiology reports at three different levels, i.e., pathological region-level, instance-level, and disease-level. Specifically, we first incorporate the instance-wise alignment module by maximizing the agreement between image-report pairs. Further, for token-wise alignment, we introduce a bidirectional cross-attention strategy to explicitly learn the matching between fine-grained visual tokens and text tokens, followed by contrastive learning to align them. More important, to leverage the high-level inter-subject relationship semantic (e.g., disease) correspondences, we design a novel cross-modal disease-level alignment paradigm to enforce the cross-modal cluster assignment consistency. Extensive experimental results on seven downstream medical image datasets covering image classification, object detection, and semantic segmentation tasks demonstrate the stable and superior performance of our framework.

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language
Fine-Grained Image-Text Alignment in Medical Imaging Enables Cyclic Image-Report Generation

Yixuan Yuan,Wenting Chen,Linlin Shen,Xiang Li

DOI: https://doi.org/10.48550/arXiv.2312.08078

Abstract:Fine-grained vision-language models (VLM) have been widely used for inter-modality local alignment between the predefined fixed patches and textual words. However, in the medical analysis, lesions exhibit varying sizes and positions, and the fixed patches may cause incomplete representations of lesions. Moreover, these methods provide explain-ability by using heatmaps to show the general potential image areas associated with texts rather than specific regions, making their explanations not explicit and specific enough. To address these issues, we propose a novel Adaptive patch-word Matching (AdaMatch) model to correlate chest X-ray (CXR) image regions with words in medical reports and apply it to CXR-report generation to provide explainabil-ity for the generation process. AdaMatch exploits the fine-grained relation between adaptive patches and words to provide explanations of specific image regions with corresponding words. To capture the abnormal regions of varying sizes and positions, we introduce the Adaptive Patch extraction (AdaPatch) module to acquire the adaptive patches for these regions adaptively. In order to provide explicit explainability for CXR-report generation task, we propose an AdaMatch-based bidirectional large language model for Cyclic CXR-report generation (AdaMatch-Cyclic). It employs the AdaMatch to obtain the keywords for CXR images and ‘keypatches’ for medical reports as hints to guide CXR-report generation. Extensive experiments on two publicly available CXR datasets prove the effectiveness of our method and its superior performance to existing methods.

Computer Science,Medicine
Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement

Weijian Huang,Cheng Li,Hao Yang,Jiarun Liu,Yong Liang,Hairong Zheng,Shanshan Wang

DOI: https://doi.org/10.1016/j.media.2024.103299

2024-09-04

Abstract:Recently, vision-language representation learning has made remarkable advancements in building up medical foundation models, holding immense potential for transforming the landscape of clinical research and medical care. The underlying hypothesis is that the rich knowledge embedded in radiology reports can effectively assist and guide the learning process, reducing the need for additional labels. However, these reports tend to be complex and sometimes even consist of redundant descriptions that make the representation learning too challenging to capture the key semantic information. This paper develops a novel iterative vision-language representation learning framework by proposing a key semantic knowledge-emphasized report refinement method. Particularly, raw radiology reports are refined to highlight the key information according to a constructed clinical dictionary and two model-optimized knowledge-enhancement metrics. The iterative framework is designed to progressively learn, starting from gaining a general understanding of the patient's condition based on raw reports and gradually refines and extracts critical information essential to the fine-grained analysis tasks. The effectiveness of the proposed framework is validated on various downstream medical image analysis tasks, including disease classification, region-of-interest segmentation, and phrase grounding. Our framework surpasses seven state-of-the-art methods in both fine-tuning and zero-shot settings, demonstrating its encouraging potential for different clinical applications.

Computer Vision and Pattern Recognition
Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation

Wenting Chen,Linlin Shen,Jingyang Lin,Jiebo Luo,Xiang Li,Yixuan Yuan

2024-06-04

Abstract:To address these issues, we propose a novel Adaptive patch-word Matching (AdaMatch) model to correlate chest X-ray (CXR) image regions with words in medical reports and apply it to CXR-report generation to provide explainability for the generation process. AdaMatch exploits the fine-grained relation between adaptive patches and words to provide explanations of specific image regions with corresponding words. To capture the abnormal regions of varying sizes and positions, we introduce the Adaptive Patch extraction (AdaPatch) module to acquire the adaptive patches for these regions adaptively. In order to provide explicit explainability for CXR-report generation task, we propose an AdaMatch-based bidirectional large language model for Cyclic CXR-report generation (AdaMatch-Cyclic). It employs the AdaMatch to obtain the keywords for CXR images and `keypatches' for medical reports as hints to guide CXR-report generation. Extensive experiments on two publicly available CXR datasets prove the effectiveness of our method and its superior performance to existing methods.

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language
Multi-modality Regional Alignment Network for Covid X-Ray Survival Prediction and Report Generation

Zhusi Zhong,Jie Li,John Sollee,Scott Collins,Harrison Bai,Paul Zhang,Terrence Healey,Michael Atalay,Xinbo Gao,Zhicheng Jiao

2024-05-23

Abstract:In response to the worldwide COVID-19 pandemic, advanced automated technologies have emerged as valuable tools to aid healthcare professionals in managing an increased workload by improving radiology report generation and prognostic analysis. This study proposes Multi-modality Regional Alignment Network (MRANet), an explainable model for radiology report generation and survival prediction that focuses on high-risk regions. By learning spatial correlation in the detector, MRANet visually grounds region-specific descriptions, providing robust anatomical regions with a completion strategy. The visual features of each region are embedded using a novel survival attention mechanism, offering spatially and risk-aware features for sentence encoding while maintaining global coherence across tasks. A cross LLMs alignment is employed to enhance the image-to-text transfer process, resulting in sentences rich with clinical detail and improved explainability for radiologist. Multi-center experiments validate both MRANet's overall performance and each module's composition within the model, encouraging further advancements in radiology report generation research emphasizing clinical interpretation and trustworthiness in AI models applied to medical studies. The code is available at

Image and Video Processing,Computer Vision and Pattern Recognition
Rethinking masked image modelling for medical image representation

Yutong Xie,Lin Gu,Tatsuya Harada,Jianpeng Zhang,Yong Xia,Qi Wu

DOI: https://doi.org/10.1016/j.media.2024.103304

2024-08-17

Abstract:Masked Image Modelling (MIM), a form of self-supervised learning, has garnered significant success in computer vision by improving image representations using unannotated data. Traditional MIMs typically employ a strategy of random sampling across the image. However, this random masking technique may not be ideally suited for medical imaging, which possesses distinct characteristics divergent from natural images. In medical imaging, particularly in pathology, disease-related features are often exceedingly sparse and localized, while the remaining regions appear normal and undifferentiated. Additionally, medical images frequently accompany reports, directly pinpointing pathological changes' location. Inspired by this, we propose Masked medical Image Modelling (MedIM), a novel approach, to our knowledge, the first research that employs radiological reports to guide the masking and restore the informative areas of images, encouraging the network to explore the stronger semantic representations from medical images. We introduce two mutual comprehensive masking strategies, knowledge-driven masking (KDM), and sentence-driven masking (SDM). KDM uses Medical Subject Headings (MeSH) words unique to radiology reports to identify symptom clues mapped to MeSH words (e.g., cardiac, edema, vascular, pulmonary) and guide the mask generation. Recognizing that radiological reports often comprise several sentences detailing varied findings, SDM integrates sentence-level information to identify key regions for masking. MedIM reconstructs images informed by this masking from the KDM and SDM modules, promoting a comprehensive and enriched medical image representation. Our extensive experiments on seven downstream tasks covering multi-label/class image classification, pneumothorax segmentation, and medical image-report analysis, demonstrate that MedIM with report-guided masking achieves competitive performance. Our method substantially outperforms ImageNet pre-training, MIM-based pre-training, and medical image-report pre-training counterparts. Codes are available at https://github.com/YtongXie/MedIM.
Radiology Report Generation with a Learned Knowledge Base and Multi-Modal Alignment

Shuxin Yang,Xian Wu,Shen Ge,Zhuozhao Zheng,S. Kevin Zhou,Li Xiao

DOI: https://doi.org/10.1016/j.media.2023.102798

IF: 10.9

2023-01-01

Medical Image Analysis

Abstract:In clinics, a radiology report is crucial for guiding a patient's treatment. However, writing radiology reports is a heavy burden for radiologists. To this end, we present an automatic, multi-modal approach for report generation from a chest x-ray. Our approach, motivated by the observation that the descriptions in radiology reports are highly correlated with specific information of the x-ray images, features two distinct modules: (i) Learned knowledge base: To absorb the knowledge embedded in the radiology reports, we build a knowledge base that can automatically distill and restore medical knowledge from textual embedding without manual labor; (ii) Multi-modal alignment: to promote the semantic alignment among reports, disease labels, and images, we explicitly utilize textual embedding to guide the learning of the visual feature space. We evaluate the performance of the proposed model using metrics from both natural language generation and clinic efficacy on the public IU-Xray and MIMIC-CXR datasets. Our ablation study shows that each module contributes to improving the quality of generated reports. Furthermore, the assistance of both modules, our approach outperforms state-of-the-art methods over almost all the metrics. Code is available at https://github.com/LX-doctorAI1/M2KT.
Cross-modal Contrastive Attention Model for Medical Report Generation.

Xiao Song,Xiaodan Zhang,Junzhong Ji,Ying Liu,Pengxu Wei

2022-01-01

Abstract:Medical report automatic generation has gained increasing interest recently as a way to help radiologists write reports more efficiently. However, this image-to-text task is rather challenging due to the typical data biases: 1) Normal physiological structures dominate the images, with only tiny abnormalities; 2) Normal descriptions accordingly dominate the reports. Existing methods have attempted to solve these problems, but they neglect to exploit useful information from similar historical cases. In this paper, we propose a novel Cross-modal Contrastive Attention (CMCA) model to capture both visual and semantic information from similar cases, with mainly two modules: a Visual Contrastive Attention Module for refining the unique abnormal regions compared to the retrieved case images; a Cross-modal Attention Module for matching the positive semantic information from the case reports. Extensive experiments on two widely-used benchmarks, IU X-Ray and MIMIC-CXR, demonstrate that the proposed model outperforms the state-of-the-art methods on almost all metrics. Further analyses also validate that our proposed model is able to improve the reports with more accurate abnormal findings and richer descriptions.

A Clinical Neurophysiology Information System based on Tcl/Tk

Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning

MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models

MCL: Multi-view Enhanced Contrastive Learning for Chest X-ray Report Generation

A Comparison of Maternal Interview and Medical Record Ascertainment of Violence among Women who had Poor Pregnancy Outcomes

MKCL: Medical Knowledge with Contrastive Learning model for radiology report generation

Radiology Reports Improve Visual Representations Learned from Radiographs

Masked Contrastive Reconstruction for Cross-modal Medical Image-Report Retrieval

MoRE: Multi-Modal Contrastive Pre-training with Transformers on X-Rays, ECGs, and Diagnostic Report

Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques

Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation

From Few to More: Scribble-based Medical Image Segmentation via Masked Context Modeling and Continuous Pseudo Labels

Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning

Fine-Grained Image-Text Alignment in Medical Imaging Enables Cyclic Image-Report Generation

Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement

Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation

Multi-modality Regional Alignment Network for Covid X-Ray Survival Prediction and Report Generation

Rethinking masked image modelling for medical image representation

Radiology Report Generation with a Learned Knowledge Base and Multi-Modal Alignment

Cross-modal Contrastive Attention Model for Medical Report Generation.