Combining natural and artificial intelligence for robust automatic anatomy segmentation: Application in neck and thorax auto‐contouring
Jayaram K. Udupa,Tiange Liu,Chao Jin,Liming Zhao,Dewey Odhner,Yubing Tong,Vibhu Agrawal,Gargi Pednekar,Sanghita Nag,Tarun Kotia,Mark Goodman,E. Paul Wileyto,Dimitris Mihailidis,John Nicholas Lukens,Abigail T. Berman,Joann Stambaugh,Tristan Lim,Rupa Chowdary,Dheeraj Jalluri,Salma K. Jabbour,Sung Kim,Meral Reyhan,Clifford G. Robinson,Wade L. Thorstad,Jehee Isabelle Choi,Robert Press,Charles B. Simone,Joe Camaratta,Steve Owens,Drew A. Torigian
DOI: https://doi.org/10.1002/mp.15854
IF: 4.506
2022-07-19
Medical Physics
Abstract:Background Automatic segmentation of 3D objects in computed tomography is challenging. Current methods, based mainly on artificial intelligence (AI) and end‐to‐end deep learning networks, are weak in garnering high‐level anatomic information, which leads to compromised efficiency and robustness. This can be overcome by incorporating natural intelligence (NI) into AI methods via computational models of human anatomic knowledge. Purpose We formulate a hybrid intelligence (HI) approach that integrates the complementary strengths of NI and AI for organ segmentation in computed tomography (CT) images and illustrate performance in the application of radiation therapy (RT) planning via multi‐site clinical evaluation. Methods The system employs five modules: (i) Body region recognition (BRR), that automatically trims a given image to a precisely‐defined target body region; (ii) NI‐based automatic anatomy recognition (AAR‐R), that performs object recognition in the trimmed image without deep learning and outputs a localized fuzzy model for each object; (iii) Deep learning‐based recognition (DL‐R), that refines the coarse recognition results of AAR‐R and outputs a stack of 2D bounding boxes (BBs) for each object; (iv) Model morphing (MM), that deforms the AAR‐R fuzzy model of each object guided by the BBs output by DL‐R; and (v) DL‐based delineation (DL‐D), that employs the object containment information provided by MM to delineate each object. NI from (ii), AI from (i), (iii), and (v), and their combination from (iv), facilitate the HI‐system. Results The HI‐system was tested on 26 organs in Neck and Thorax body regions on CT images obtained prospectively from 464 patients in a study involving four RT centers. Data sets from one separate independent institution involving 125 patients were employed in training/model building for each of the two body regions, while 104 and 110 data sets from the 4 RT centers were utilized for testing on Neck and Thorax, respectively. In the testing data sets, 83% of the images had limitations such as streak artifacts, poor contrast, shape distortion, pathology, or implants. The contours output by the HI‐system were compared to contours drawn in clinical practice at the 4 RT centers by utilizing an independently established ground truth (GT) set of contours as reference. Three sets of measures were employed: accuracy via Dice Coefficient (DC) and Hausdorff boundary Distance (HD), subjective clinical acceptability via a blinded reader study, and efficiency by measuring human time saved in contouring by the HI‐system. Overall, the HI‐system achieved a mean DC of 0.78 and 0.87 and a mean HD of 2.22 mm and 4.53 mm for Neck and Thorax, respectively. It significantly outperformed clinical contouring in accuracy and saved overall 70% of human time over clinical contouring time, while acceptability scores varied significantly from site to site for both auto‐contours and clinically drawn contours. Conclusions The HI‐system is observed to behave like an expert human in robustness in the contouring task but vastly more efficiently. It seems to use NI help where image information alone will not suffice to decide, first for the correct localization of the object and then for the precise delineation of the boundary. This article is protected by copyright. All rights reserved
radiology, nuclear medicine & medical imaging