Abstract:Foundation models for interactive segmentation in 2D natural images and videos have sparked significant interest in building 3D foundation models for medical imaging. However, the domain gaps and clinical use cases for 3D medical imaging require a dedicated model that diverges from existing 2D solutions. Specifically, such foundation models should support a full workflow that can actually reduce human effort. Treating 3D medical images as sequences of 2D slices and reusing interactive 2D foundation models seems straightforward, but 2D annotation is too time-consuming for 3D tasks. Moreover, for large cohort analysis, it's the highly accurate automatic segmentation models that reduce the most human effort. However, these models lack support for interactive corrections and lack zero-shot ability for novel structures, which is a key feature of "foundation". While reusing pre-trained 2D backbones in 3D enhances zero-shot potential, their performance on complex 3D structures still lags behind leading 3D models. To address these issues, we present VISTA3D, Versatile Imaging SegmenTation and Annotation model, that targets to solve all these challenges and requirements with one unified foundation model. VISTA3D is built on top of the well-established 3D segmentation pipeline, and it is the first model to achieve state-of-the-art performance in both 3D automatic (supporting 127 classes) and 3D interactive segmentation, even when compared with top 3D expert models on large and diverse benchmarks. Additionally, VISTA3D's 3D interactive design allows efficient human correction, and a novel 3D supervoxel method that distills 2D pretrained backbones grants VISTA3D top 3D zero-shot performance. We believe the model, recipe, and insights represent a promising step towards a clinically useful 3D foundation model. Code and weights are publicly available at <a class="link-external link-https" href="https://github.com/Project-MONAI/VISTA" rel="external noopener nofollow">this https URL</a>.

IsoExplorer: an Isosurface-Driven Framework for 3D Shape Analysis of Biomedical Volume Data

Intelligent Volume Visualization for Medical Datasets

Structure-preserving visualization for single-cell RNA-Seq profiles using deep manifold transformation with batch-correction

Surface Carving-Based Automatic Volume Data Reduction

Local Deep Feature Learning Framework for 3D Shape.

An Automatic Surface Extraction for Volume Visualization

Efficient Sparse Shape Composition with Its Applications in Biomedical Image Analysis: an Overview

Novel Iso-surface Reconstruction Algorithm for Numerical Analysis from Medical Volume Data

Intrinsic spin images: a subspace decomposition approach to understanding 3D deformable shapes

Surface Reconstruction from Sparse and Mutually Intersected Contours for Freehand 3D Ultrasound Using Variational Method

4D-Explorer: A visual software for 4D-STEM data processing and image reconstruction

Surface Reconstruction from Sparse & Arbitrarily Oriented Contours in Freehand 3D Ultrasound.

Deep Medial Voxels: Learned Medial Axis Approximations for Anatomical Shape Modeling

Variational Approach To Reconstruct Surface From Sparse & Nonparallel Contours In Freehand 3d Ultrasound Imaging

Identifying Nearly Equally Spaced Isosurfaces for Volumetric Data Sets

Functional Data Analysis and Visualisation of Three-dimensional Surface Shape

Visualization of Multiple Anatomical Structures with Explicit Isosurface Manipulation

VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging

Guide3D: A Bi-planar X-ray Dataset for 3D Shape Reconstruction

Interactive Occlusion-Free System for Accessible Volume Exploration

A geometry-informed deep learning framework for ultra-sparse 3D tomographic image reconstruction