Abstract:Background and objective. Image segmentation is an essential component in medical image analysis. The case of 3D images such as MRI is particularly challenging and time consuming. Interactive or semi-automatic methods are thus highly desirable. However, existing methods do not exploit the typical sequentiality of real user interactions. This is due to the interaction memory used in these systems, which discards ordering. In contrast, we argue that the order of the user corrections should be used for training and lead to performance improvements. Methods. We contribute to solving this problem by proposing a general multi-class deep learning-based interactive framework for image segmentation, which embeds a base network in a user interaction loop with a user feedback memory. We propose to model the memory explicitly as a sequence of consecutive system states, from which the features can be learned, generally learning from the segmentation refinement process. Training is a major difficulty owing to the network's input being dependent on the previous output. We adapt the network to this loop by introducing a virtual user in the training process, modelled by dynamically simulating the iterative user feedback. Results. We evaluated our framework against existing methods on the complex task of multi-class semantic instance female pelvis MRI segmentation with 5 classes, including up to 27 tumour instances, using a segmentation dataset collected in our hospital, and on liver and pancreas CT segmentation, using public datasets. We conducted a user evaluation, involving both senior and junior medical personnel in matching and adjacent areas of expertise. We observed an annotation time reduction with 5'56" for our framework against 25' on average for classical tools. We systematically evaluated the influence of the number of clicks on the segmentation accuracy. A single interaction round our framework outperforms existing automatic systems with a comparable setup. We provide an ablation study and show that our framework outperforms existing interactive systems. Conclusions. Our framework largely outperforms existing systems in accuracy, with the largest impact on the smallest, most difficult classes, and drastically reduces the average user segmentation time with fast inference at 47.2±6.2 ms per image.

Real-time 3D interactive segmentation of echocardiographic data through user-based deformation of B-spline explicit active surfaces

Automatic Spinal Ultrasound Image Segmentation and Deployment for Real-time Spine Volumetric Reconstruction

An effective and open source interactive 3D medical image segmentation solution

DEEPBEAS3D: Deep Learning and B-Spline Explicit Active Surfaces

Spatiotemporal Strategies for Joint Segmentation and Motion Tracking from Cardiac Image Sequences

Segmentation of Ultrasound Images with Interactive B-SplineSnakes and Its Application

A novel tool for segmenting 3D medical images based on generalized cylinders and active surfaces

A deep learning-based interactive medical image segmentation framework with sequential memory

Simultaneous segmentation and motion recovery in 3d cardiac image analysis

Automatic segmentation and implicit surface representation of dynamic cardiac data

Interactive Deep Editing Framework for Medical Image Segmentation.

A hybrid framework for 3D medical image segmentation

An efficient interactive multi-label segmentation tool for 2D and 3D medical images using fully connected conditional random field

A dynamic interactive learning framework for automated 3D medical image segmentation

Interactive Object Segmentation in 3D Point Clouds

DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation

MIDeepSeg: Minimally interactive segmentation of unseen objects from medical images using deep learning

From Sparse to Precise: A Practical Editing Approach for Intracardiac Echocardiography Segmentation

Computing Synthetic Echocardiography Volumes for Automatic Validation of 3d Segmentation Results

Cross Modality Deformable Segmentation Using Hierarchical Clustering and Learning.

Multi-task Learning for Motion Analysis and Segmentation in 3D Echocardiography