Abstract:As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.

Medical computer vision, virtual reality and robotics

3D Visualization in Digital Medicine Using XR Technology

Tracking and mapping in medical computer vision: A review

Computer vision and machine learning for medical image analysis: recent advances, challenges, and way forward

VR interaction techniques for medical imaging applications

From medical imaging to virtual reality for archaeology

New Approach to Accelerated Image Annotation by Leveraging Virtual Reality and Cloud Computing

AI in Medical Imaging Informatics: Current Challenges and Future Directions

3d Medical Imaging And Augmented Reality For Image-Guided Surgery

Research challenges and emerging futuristic evolution for 3D medical image processing

Immersive Anatomical Scenes that Enable Multiple Users to Occupy the Same Virtual Space: A Tool for Surgical Planning and Education

Processing and Visualization of Medical Data in a Multiuser Environment Using Artificial Intelligence

Artificial Intelligence-based algorithms in medical image scan seg-mentation and intelligent visual-content generation -- a concise overview

Review:Visualization of Three-Dimensional Medical Images

A Review of Three-Dimensional Medical Image Visualization

Medical image computing and computer-aided medical interventions applied to soft tissues. Work in progress in urology

Automatic Tool Landmark Detection for Stereo Vision in Robot-Assisted Retinal Surgery

Computer Vision in the Operating Room: Opportunities and Caveats

Artificial Intelligence-Based Algorithms in Medical Image Scan Segmentation and Intelligent Visual Content Generation—A Concise Overview