End-to-End Breast Cancer Radiotherapy Planning via LMMs with Consistency Embedding

Kwanyoung Kim,Yujin Oh,Sangjoon Park,Hwa Kyung Byun,Joongyo Lee,Jin Sung Kim,Yong Bae Kim,Jong Chul Ye
2024-07-02
Abstract:Recent advances in AI foundation models have significant potential for lightening the clinical workload by mimicking the comprehensive and multi-faceted approaches used by medical professionals. In the field of radiation oncology, the integration of multiple modalities holds great importance, so the opportunity of foundational model is abundant. Inspired by this, here we present RO-LMM, a multi-purpose, comprehensive large multimodal model (LMM) tailored for the field of radiation oncology. This model effectively manages a series of tasks within the clinical workflow, including clinical context summarization, radiation treatment plan suggestion, and plan-guided target volume segmentation by leveraging the capabilities of LMM. In particular, to perform consecutive clinical tasks without error accumulation, we present a novel Consistency Embedding Fine-Tuning (CEFTune) technique, which boosts LMM's robustness to noisy inputs while preserving the consistency of handling clean inputs. We further extend this concept to LMM-driven segmentation framework, leading to a novel Consistency Embedding Segmentation~(CESEG) techniques. Experimental results including multi-centre validation confirm that our RO-LMM with CEFTune and CESEG results in promising performance for multiple clinical tasks with generalization capabilities.
Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning
What problem does this paper attempt to address?
The purpose of this paper is to address the automation problem in radiotherapy treatment planning for breast cancer in radiation oncology. The study proposes a large-scale multimodal model called RO-LMM, which is capable of handling multiple tasks in the clinical workflow, including clinical context summarization, radiation treatment plan suggestion, and target volume segmentation for planning guidance. To prevent error accumulation in consecutive tasks, a new training technique called Consistency Embedding Fine-tuning (CEFTune) is proposed in the paper to enhance the model's robustness to noisy inputs while maintaining consistent processing for clean inputs. Additionally, they extend this concept to image segmentation tasks and develop Noise Embedding Segmentation (NESEG) and Consistency Embedding Segmentation (CESEG) techniques to reduce error propagation and improve the model's generalization ability. Experimental results demonstrate that RO-LMM performs well in multi-center validation, outperforming baseline methods, and is suitable for radiation treatment of breast cancer.