Abstract:To mitigate the challenges of operating through narrow incisions under image guidance, there is a desire to develop intelligent systems that assist decision making and spatial reasoning in minimally invasive surgery (MIS). In this context, machine learning-based systems for interventional image analysis are receiving considerable attention because of their flexibility and the opportunity to provide immediate, informative feedback to clinicians. It is further believed that learning-based image analysis may eventually form the foundation for semior fully automated delivery of surgical treatments. A significant bottleneck in developing such systems is the availability of annotated images with sufficient variability to train generalizable models, particularly the most recently favored deep convolutional neural networks or transformer architectures. A popular alternative to acquiring and manually annotating data from the clinical practice is the simulation of these data from human-based models. Simulation has many advantages, including the avoidance of ethical issues, precisely controlled environments, and the scalability of data collection. Here, we survey recent work that relies on in silico training of learning-based MIS systems, in which data are generated via computational simulation. For each imaging modality, we review available simulation tools in terms of compute requirements, image quality, and usability, as well as their applications for training intelligent systems. We further discuss open challenges for simulation-based development of MIS systems, such as the need for integrated imaging and physical modeling for non-optical modalities, as well as generative patient models not dependent on underlying CT, MRI, or other patient data. In conclusion, as the capabilities of in silico training mature, with respect to sim-to-real transfer, computational efficiency, and degree of control, they are contributing toward the next generation of intelligent surgical systems.

The realization of medical devices for precision surgery – development and implementation of 'stop-and-go' imaging technologies

The current status and future prospects for molecular imaging-guided precision surgery

Analysis of Intraoperative Microscopy Imaging Techniques and Their Future Applications

In Vivo And In Situ Image Guidance And Modelling In Robotic Assisted Surgery

3d Medical Imaging And Augmented Reality For Image-Guided Surgery

Emerging Intraoperative Imaging Modalities to Improve Surgical Precision

Computerized Medical Imaging and Graphics

New Technologies Supporting Surgical Intervenltions and Training of Surgical Skills - A Look at Projects in Europe Supporting Minimally Invasive Techniques

From Medical Images to Minimally Invasive Intervention: Computer Assistance for Robotic Surgery

State of the art medical devices for fluorescence-guided surgery (FGS): technical review and future developments

Pioneering Patient-Specific Approaches for Precision Surgery Using Imaging and Virtual Reality

Smart Sensing for Surgery: From Tethered Devices to Wearables and Implantables

State of the Art in Immersive Interactive Technologies for Surgery Simulation: A Review and Prospective

Computer-aided design and manufacturing of surgical templates and their clinical applications: a review

State of the Art and Future Opportunities in MRI-Guided Robot-Assisted Surgery and Interventions

The Principles And Role Of Medical Imaging In Surgery

Chapter 13 - Medical Robotics for Cellular and Molecular Imaging

Onward to Better Surgery - the Critical Need for Improved Ex Vivo Testing and Training Methods

In Silico simulation: A key enabling technology for next-generation intelligent surgical systems

The Role of Technology in Minimally Invasive Surgery: State of the Art, Recent Developments and Future Directions

Real-time Virtual Intraoperative CT for Image Guided Surgery