Abstract:Medical image interpretation is central to most clinical applications such as disease diagnosis, treatment planning, and prognostication. In clinical practice, radiologists examine medical images and manually compile their findings into reports, which can be a time-consuming process. Automated approaches to radiology report generation, therefore, can reduce radiologist workload and improve efficiency in the clinical pathway. While recent deep-learning approaches for automated report generation from medical images have seen some success, most studies have relied on image-derived features alone, ignoring non-imaging patient data. Although a few studies have included the word-level contexts along with the image, the use of patient demographics is still unexplored. This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information, to synthesise patient-specific radiology reports. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information, to synthesise full-text radiology reports. Data from two public databases were used to train and evaluate the proposed approach. CXRs and reports were extracted from the MIMIC-CXR database and combined with corresponding patients' data MIMIC-IV. Based on the evaluation metrics used including patient demographic information was found to improve the quality of reports generated using the proposed approach, relative to a baseline network trained using CXRs alone. The proposed approach shows potential for enhancing radiology report generation by leveraging rich patient metadata and combining semantic text embeddings derived thereof, with medical image-derived visual features.

Embracing Uniqueness: Generating Radiology Reports via a Transformer with Graph-based Distinctive Attention.

An Inclusive Task-Aware Framework for Radiology Report Generation

Generating Radiology Reports via Memory-driven Transformer

Automated Radiographic Report Generation Purely on Transformer: A Multicriteria Supervised Approach

Generating Radiology Reports Via Multi-feature Optimization Transformer

Radiology Report Generation Using Transformers Conditioned with Non-imaging Data

Work like a doctor: Unifying scan localizer and dynamic generator for automated computed tomography report generation

A Medical Semantic-Assisted Transformer for Radiographic Report Generation

CSAMDT: Conditional Self Attention Memory-Driven Transformers for Radiology Report Generation from Chest X-Ray

Radiology Report Generation for Rare Diseases via Few-shot Transformer.

TSGET: Two-Stage Global Enhanced Transformer for Automatic Radiology Report Generation

Act Like a Radiologist: Radiology Report Generation across Anatomical Regions

Prior-RadGraphFormer: A Prior-Knowledge-Enhanced Transformer for Generating Radiology Graphs from X-Rays

The Long-Term Memory Transformer with Multimodal Fusion for Radiology Report Generation

Radiology Report Generation via Structured Knowledge-Enhanced Multi-modal Attention and Contrastive Learning.

Interactive dual-stream contrastive learning for radiology report generation

MATNet: Exploiting Multi-Modal Features for Radiology Report Generation.

An efficient but effective writer: Diffusion-based semi-autoregressive transformer for automated radiology report generation

MeFormer: Generating Radiology Reports via Memory Enhanced Pretraining Transformer

Prior Guided Transformer for Accurate Radiology Reports Generation

Semi-supervised Medical Report Generation via Graph-guided Hybrid Feature Consistency