Abstract:Modeling and recognition of surgical activities poses an interesting research problem. Although a number of recent works studied automatic recognition of surgical activities, generalizability of these works across different tasks and different datasets remains a challenge. We introduce a modality that is robust to scene variation, and that is able to infer part information such as orientational and relative spatial relationships. The proposed modality is based on spatial temporal graph representations of surgical tools in videos, for surgical activity recognition. To explore its effectiveness, we model and recognize surgical gestures with the proposed modality. We construct spatial graphs connecting the joint pose estimations of surgical tools. Then, we connect each joint to the corresponding joint in the consecutive frames forming inter-frame edges representing the trajectory of the joint over time. We then learn hierarchical spatial temporal graph representations using Spatial Temporal Graph Convolutional Networks (ST-GCN). Our experiments show that learned spatial temporal graph representations perform well in surgical gesture recognition even when used individually. We experiment with the Suturing task of the JIGSAWS dataset where the chance baseline for gesture recognition is 10%. Our results demonstrate 68% average accuracy which suggests a significant improvement. Learned hierarchical spatial temporal graph representations can be used either individually, in cascades or as a complementary modality in surgical activity recognition, therefore provide a benchmark for future studies. To our knowledge, our paper is the first to use spatial temporal graph representations of surgical tools, and pose-based skeleton representations in general, for surgical activity recognition.

Concept Graph Neural Networks for Surgical Video Understanding

Towards multimodal graph neural networks for surgical instrument anticipation

Aggregating Long-Term Context for Learning Laparoscopic and Robot-Assisted Surgical Workflows

Dynamic Scene Graph Representation for Surgical Video

Hypergraph-Transformer (HGT) for Interactive Event Prediction in Laparoscopic and Robotic Surgery

Towards Graph Representation Learning Based Surgical Workflow Anticipation

Abstracting Deep Neural Networks into Concept Graphs for Concept Level Interpretability

Towards Generalizable Surgical Activity Recognition Using Spatial Temporal Graph Convolutional Networks

Learning and Reasoning with the Graph Structure Representation in Robotic Surgery

SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow Prediction

Adaptive Graph Learning from Spatial Information for Surgical Workflow Anticipation

Optimizing latent graph representations of surgical scenes for unseen domain generalization

Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms

Biomimetic Incremental Domain Generalization with a Graph Network for Surgical Scene Understanding

Advancing surgical VQA with scene graph knowledge

A spatio-temporal network for video semantic segmentation in surgical videos

Multi-Granular Semantic Analysis Based on Nasal Endoscopic Video

SurGNN: Explainable visual scene understanding and assessment of surgical skill using graph neural networks

Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures

Concept Graphs: A Novel Approach for Textual Analysis of Medical Documents

Surgical Video Workflow Analysis Via Visual-Language Learning