Recent Progress in Transformer-based Medical Image Analysis

Zhaoshan Liu,Qiujie Lv,Ziduo Yang,Yifan Li,Chau Hung Lee,Lei Shen

DOI: https://doi.org/10.1016/j.compbiomed.2023.107268

2023-07-25

Abstract:The transformer is primarily used in the field of natural language processing. Recently, it has been adopted and shows promise in the computer vision (CV) field. Medical image analysis (MIA), as a critical branch of CV, also greatly benefits from this state-of-the-art technique. In this review, we first recap the core component of the transformer, the attention mechanism, and the detailed structures of the transformer. After that, we depict the recent progress of the transformer in the field of MIA. We organize the applications in a sequence of different tasks, including classification, segmentation, captioning, registration, detection, enhancement, localization, and synthesis. The mainstream classification and segmentation tasks are further divided into eleven medical image modalities. A large number of experiments studied in this review illustrate that the transformer-based method outperforms existing methods through comparisons with multiple evaluation metrics. Finally, we discuss the open challenges and future opportunities in this field. This task-modality review with the latest contents, detailed information, and comprehensive comparison may greatly benefit the broad MIA community.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily explores the advancements and advantages of Transformer in the field of Medical Image Analysis (MIA). Specifically:

1. **Application of Transformer Technology in MIA**:
- Introduces the core component of Transformer (attention mechanism) and its detailed structure.
- Provides an overview of the latest advancements of Transformer in medical image analysis, categorized by different tasks including classification, segmentation, captioning, registration, detection, enhancement, localization, and synthesis.

2. **Summary of Applications under Different Tasks**:
- Further subdivides the mainstream classification and segmentation tasks, categorizing them into 11 different medical imaging modalities, including MRI, CT, X-ray, microscopy, endoscopy, ultrasound, dermoscopy, fundus imaging (DFI), camera, positron emission tomography (PET), and optical coherence tomography (OCT).

3. **Performance Comparison and Discussion**:
- Provides a large number of experimental results, showing that Transformer-based methods outperform existing methods under various evaluation metrics.
- Discusses the current challenges and future research opportunities, offering comprehensive references for researchers in the field of medical image analysis.

Through these contents, the paper aims to help researchers better understand the latest advancements and potential applications of Transformer in the MIA field, thereby promoting the development of this domain.