Abstract:Modeling and recognition of surgical activities poses an interesting research problem. Although a number of recent works studied automatic recognition of surgical activities, generalizability of these works across different tasks and different datasets remains a challenge. We introduce a modality that is robust to scene variation, and that is able to infer part information such as orientational and relative spatial relationships. The proposed modality is based on spatial temporal graph representations of surgical tools in videos, for surgical activity recognition. To explore its effectiveness, we model and recognize surgical gestures with the proposed modality. We construct spatial graphs connecting the joint pose estimations of surgical tools. Then, we connect each joint to the corresponding joint in the consecutive frames forming inter-frame edges representing the trajectory of the joint over time. We then learn hierarchical spatial temporal graph representations using Spatial Temporal Graph Convolutional Networks (ST-GCN). Our experiments show that learned spatial temporal graph representations perform well in surgical gesture recognition even when used individually. We experiment with the Suturing task of the JIGSAWS dataset where the chance baseline for gesture recognition is 10%. Our results demonstrate 68% average accuracy which suggests a significant improvement. Learned hierarchical spatial temporal graph representations can be used either individually, in cascades or as a complementary modality in surgical activity recognition, therefore provide a benchmark for future studies. To our knowledge, our paper is the first to use spatial temporal graph representations of surgical tools, and pose-based skeleton representations in general, for surgical activity recognition.

Adaptive Graph Learning from Spatial Information for Surgical Workflow Anticipation

Towards Graph Representation Learning Based Surgical Workflow Anticipation

SWAG: Long-term Surgical Workflow Prediction with Generative-based Anticipation

Towards multimodal graph neural networks for surgical instrument anticipation

Graph-Based Spatial Reasoning for Tracking Landmarks in Dynamic Laparoscopic Environments

SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow Prediction

Hypergraph-Transformer (HGT) for Interactive Event Prediction in Laparoscopic and Robotic Surgery

Rethinking Anticipation Tasks: Uncertainty-aware Anticipation of Sparse Surgical Instrument Usage for Context-aware Assistance

Aggregating Long-Term Context for Learning Laparoscopic and Robot-Assisted Surgical Workflows

SUPR-GAN: SUrgical PRediction GAN for Event Anticipation in Laparoscopic and Robotic Surgery

Learning and Reasoning with the Graph Structure Representation in Robotic Surgery

Laparoscopic Image-based Critical Action Recognition and Anticipation with Explainable Features

Future Frame Prediction for Robot-assisted Surgery

Towards Generalizable Surgical Activity Recognition Using Spatial Temporal Graph Convolutional Networks

Efficient and Accurate Mapping of Subsurface Anatomy via Online Trajectory Optimization for Robot Assisted Surgery

Latent Graph Representations for Critical View of Safety Assessment

LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms

ST-ITEF: Spatio-Temporal Intraoperative Task Estimating Framework to recognize surgical phase and predict instrument path based on multi-object tracking in keratoplasty

Dynamic Scene Graph Representation for Surgical Video

daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgery

Deep Learning for Surgical Workflow Analysis: a Survey of Progresses, Limitations, and Trends