Abstract:Minimally Invasive Surgery (MIS) techniques have gained rapid popularity among surgeons since they offer significant clinical benefits including reduced recovery time and diminished post-operative adverse effects. However, conventional endoscopic systems output monocular video which compromises depth perception, spatial orientation and field of view. Suturing is one of the most complex tasks performed under these circumstances. Key components of this tasks are the interplay between needle holder and the surgical needle. Reliable 3D localization of needle and instruments in real time could be used to augment the scene with additional parameters that describe their quantitative geometric relation, e.g. the relation between the estimated needle plane and its rotation center and the instrument. This could contribute towards standardization and training of basic skills and operative techniques, enhance overall surgical performance, and reduce the risk of complications. The paper proposes an Augmented Reality environment with quantitative and qualitative visual representations to enhance laparoscopic training outcomes performed on a silicone pad. This is enabled by a multi-task supervised deep neural network which performs multi-class segmentation and depth map prediction. Scarcity of labels has been conquered by creating a virtual environment which resembles the surgical training scenario to generate dense depth maps and segmentation maps. The proposed convolutional neural network was tested on real surgical training scenarios and showed to be robust to occlusion of the needle. The network achieves a dice score of 0.67 for surgical needle segmentation, 0.81 for needle holder instrument segmentation and a mean absolute error of 6.5 mm for depth estimation.

Rethinking Anticipation Tasks: Uncertainty-aware Anticipation of Sparse Surgical Instrument Usage for Context-aware Assistance

Adaptive Graph Learning from Spatial Information for Surgical Workflow Anticipation

Towards Graph Representation Learning Based Surgical Workflow Anticipation

Robotic scrub nurse to anticipate surgical instruments based on real-time laparoscopic video analysis

Towards multimodal graph neural networks for surgical instrument anticipation

Real-time image-based instrument classification for laparoscopic surgery

Predicting the Timing of Camera Movements From the Kinematics of Instruments in Robotic-Assisted Surgery Using Artificial Neural Networks

Laparoscopic Image-based Critical Action Recognition and Anticipation with Explainable Features

SUPR-GAN: SUrgical PRediction GAN for Event Anticipation in Laparoscopic and Robotic Surgery

WeakSurg: Weakly supervised surgical instrument segmentation using temporal equivariance and semantic continuity

Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery

Uncertainty-Aware Anticipation of Activities

Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos

Aggregating Long-Term Context for Learning Laparoscopic and Robot-Assisted Surgical Workflows

Untrimmed Action Anticipation

Surgical Phase Recognition in Inguinal Hernia Repair—AI-Based Confirmatory Baseline and Exploration of Competitive Models

Future Frame Prediction for Robot-assisted Surgery

Unsupervised Temporal Video Segmentation as an Auxiliary Task for Predicting the Remaining Surgery Duration

Towards Augmented Reality-based Suturing in Monocular Laparoscopic Training

Synthetic and Real Inputs for Tool Segmentation in Robotic Surgery

Gesture Recognition in Robotic Surgery With Multimodal Attention