Joint Liver and Hepatic Lesion Segmentation in MRI using a Hybrid CNN with Transformer Layers

Georg Hille,Shubham Agrawal,Pavan Tummala,Christian Wybranski,Maciej Pech,Alexey Surov,Sylvia Saalfeld
DOI: https://doi.org/10.48550/arXiv.2201.10981
2023-03-22
Abstract:Deep learning-based segmentation of the liver and hepatic lesions therein steadily gains relevance in clinical practice due to the increasing incidence of liver cancer each year. Whereas various network variants with overall promising results in the field of medical image segmentation have been successfully developed over the last years, almost all of them struggle with the challenge of accurately segmenting hepatic lesions in magnetic resonance imaging (MRI). This led to the idea of combining elements of convolutional and transformer-based architectures to overcome the existing limitations. This work presents a hybrid network called SWTR-Unet, consisting of a pretrained ResNet, transformer blocks as well as a common Unet-style decoder path. This network was primarily applied to single-modality non-contrast-enhanced liver MRI and additionally to the publicly available computed tomography (CT) data of the liver tumor segmentation (LiTS) challenge to verify the applicability on other modalities. For a broader evaluation, multiple state-of-the-art networks were implemented and applied, ensuring a direct comparability. Furthermore, correlation analysis and an ablation study were carried out, to investigate various influencing factors on the segmentation accuracy of the presented method. With Dice scores of averaged 98+-2% for liver and 81+-28% lesion segmentation on the MRI dataset and 97+-2% and 79+-25%, respectively on the CT dataset, the proposed SWTR-Unet proved to be a precise approach for liver and hepatic lesion segmentation with state-of-the-art results for MRI and competing accuracy in CT imaging. The achieved segmentation accuracy was found to be on par with manually performed expert segmentations as indicated by inter-observer variabilities for liver lesion segmentation. In conclusion, the presented method could save valuable time and resources in clinical practice.
Image and Video Processing,Computer Vision and Pattern Recognition,Machine Learning
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the accurate segmentation of the liver and its intra - hepatic lesions in magnetic resonance imaging (MRI). With the increasing incidence of liver cancer every year, the segmentation of the liver and intra - hepatic lesions based on deep learning has become more and more important in clinical practice. However, although most of the existing network variants have generally achieved satisfactory results in the field of medical image segmentation, they still face challenges in accurately segmenting intra - hepatic lesions in MRI. Therefore, this study aims to overcome the limitations of existing methods by combining convolutional neural network (CNN) and Transformer - based architecture elements, and proposes a new hybrid network architecture, namely SWTR - Unet, to improve the segmentation accuracy. Specifically, the main objectives of the paper include: 1. **Develop a fully - automatic deep - learning model** for jointly segmenting the liver and intra - hepatic lesions in MRI. This model can achieve expert - level segmentation accuracy and is applicable to clinical MRI and CT data. 2. **Evaluate the performance of the proposed SWTR - Unet in different modalities**, including single - modality non - contrast - enhanced liver MRI and publicly available CT data (from the LiTS challenge), and verify its applicability in other modalities. 3. **Compare with the existing state - of - the - art networks** to ensure the superiority or competitiveness of the proposed method. 4. **Through correlation analysis and ablation studies**, explore various factors that affect segmentation accuracy, such as the number of skip connections, the number of Transformer layers, and the influence of lesion size and shape on the segmentation results. In conclusion, this study aims to provide an efficient and accurate method for segmenting the liver and intra - hepatic lesions, to support radiologists in tumor staging and treatment decision - making, and save precious time and resources.