Abstract:Background and objective: Medical imaging techniques are widely employed in disease diagnosis and treatment. A readily available medical report can be a useful tool in assisting an expert for investigating the patient's health. A radiologist can benefit from an automatic medical image to radiological report translation system while preparing a final report. Previous attempts on automatic medical report generation task includes image captioning algorithms without taking domain-specific visual and textual contents into account, thus arises the question about credibility of generated report. Methods: In this work, a novel Adaptive Multilevel Multi-Attention (AMLMA) approach is proposed by offering domain-specific visual-textual knowledge to generate a thorough and believable radiological report for any view of a human chest X-ray image. The proposed approach leverages the encoder-decoder framework incorporated with multiple adaptive attention mechanisms. The potential of a convolutional neural network (CNN) with residual attention module (RAM) is demonstrated as a strong visual encoder for multi-label abnormality detection. The multilevel visual features (local and global) are extracted from proposed visual encoder to retrieve regional-level and abstract-level radiology-based semantic information. The Word2Vec and FastText word embeddings are trained on medical reports to acquire radiological knowledge and further used as textual encoders, feeding as input to Bi-directional Long Short Term Memory (Bi-LSTM) network to learn the co-relationship between medical terminologies in radiological reports. The AMLMA employs a weighted multilevel association of adaptive visual-semantic attention and visual-based linguistic attention mechanisms. This association of adaptive attention is exploited as a decoder and produces significant improvements in the report generation task. Results: The proposed approach is evaluated on a publicly available Indiana University chest X-ray (IU-CXR) dataset. The CNN with RAM shows the significant improvement in recall (0.4423), precision (0.1803) and F1-score (0.2551) for prediction of multiple abnormalities in X-ray image. The results of language generation metrics for proposed variants were acquired using the COCO-caption evaluation Application Program Interface (API). The trained embeddings with AMLMA model generates the convincing radiology report and outperform state-of-the-art (SOTA) approaches with high evaluation metrics scores for Bleu-4 (0.172), Meteor (0.247), Rouge_L (0.376) and CIDEr (0.381). In addition, a new "Unique Index" (UI) statistic is introduced to highlight the model's ability for generating unique reports. Conclusion: The overall architecture aids to the understanding of various X-ray image views and generating the relevant normal and abnormal radiography statements. The proposed model is emphasized on multi-level visual-textual knowledge with adaptive attention mechanism to balance visual and linguistic information for the generation of admissible radiology report.

From technical to understandable: Artificial Intelligence Large Language Models improve the readability of knee radiology reports

Decoding Radiology Reports: Artificial Intelligence-Large Language Models Can Improve the Readability of Hand and Wrist Orthopedic Radiology Reports

From jargon to clarity: Improving the readability of foot and ankle radiology reports with an artificial intelligence large language model

Translating musculoskeletal radiology reports into patient-friendly summaries using ChatGPT-4

Large Language Models for Simplified Interventional Radiology Reports: A Comparative Analysis

Evaluation of large language models performance against humans for summarizing MRI knee radiology reports: A feasibility study

Generating colloquial radiology reports with large language models

Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports

Patient-centered radiology reports with generative artificial intelligence: adding value to radiology reporting

Patient Centric Summarization of Radiology Findings using Large Language Models

Constructing a Large Language Model to Generate Impressions from Findings in Radiology Reports

Evaluation of Radiology Residents' Reporting Skills Using Large Language Models: An Observational Study

Improving Automating Quality Control in Radiology: Leveraging Large Language Models to Extract Correlative Findings in Radiology and Operative Reports

Collaboration between clinicians and vision–language models in radiology report generation

Evaluating Artificial Intelligence's Role in Teaching the Reporting and Interpretation of Computed Tomographic Angiography for Preoperative Planning of the Deep Inferior Epigastric Artery Perforator Flap

LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation

Learning to Generate Radiology Findings from Impressions Based on Large Language Model

Large language models for structured reporting in radiology: past, present, and future

Translating medical image to radiological report: Adaptive multilevel multi-attention approach

Exploring Multimodal Large Language Models for Radiology Report Error-checking

Improving Radiology Report Conciseness and Structure via Local Large Language Models