Abstract:Modern operating room is becoming increasingly complex, requiring innovative intra-operative support systems. While the focus of surgical data science has largely been on video analysis, integrating surgical computer vision with language capabilities is emerging as a necessity. Our work aims to advance Visual Question Answering (VQA) in the surgical context with scene graph knowledge, addressing two main challenges in the current surgical VQA systems: removing question-condition bias in the surgical VQA dataset and incorporating scene-aware reasoning in the surgical VQA model design. First, we propose a Surgical Scene Graph-based dataset, SSG-QA, generated by employing segmentation and detection models on publicly available datasets. We build surgical scene graphs using spatial and action information of instruments and anatomies. These graphs are fed into a question engine, generating diverse QA pairs. Our SSG-QA dataset provides a more complex, diverse, geometrically grounded, unbiased, and surgical action-oriented dataset compared to existing surgical VQA datasets. We then propose SSG-QA-Net, a novel surgical VQA model incorporating a lightweight Scene-embedded Interaction Module (SIM), which integrates geometric scene knowledge in the VQA model design by employing cross-attention between the textual and the scene features. Our comprehensive analysis of the SSG-QA dataset shows that SSG-QA-Net outperforms existing methods across different question types and complexities. We highlight that the primary limitation in the current surgical VQA systems is the lack of scene knowledge to answer complex queries. We present a novel surgical VQA dataset and model and show that results can be significantly improved by incorporating geometric scene features in the VQA model design. The source code and the dataset will be made publicly available at: <a class="link-external link-https" href="https://github.com/CAMMA-public/SSG-QA" rel="external noopener nofollow">this https URL</a>

Towards Holistic Surgical Scene Understanding

Pixel-Wise Recognition for Holistic Surgical Scene Understanding

MuST: Multi-Scale Transformers for Surgical Phase Recognition

SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge

DeepPhase: Surgical Phase Recognition in CATARACTS Videos

Quantification of Robotic Surgeries with Vision-Based Deep Learning

Surgical Phase Recognition in Inguinal Hernia Repair—AI-Based Confirmatory Baseline and Exploration of Competitive Models

Laparoscopic Image-based Critical Action Recognition and Anticipation with Explainable Features

SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow Prediction

COMPASS: A Formal Framework and Aggregate Dataset for Generalized Surgical Procedure Modeling

LoViT: Long Video Transformer for surgical phase recognition

Advancing Surgical VQA with Scene Graph Knowledge

Friends Across Time: Multi-Scale Action Segmentation Transformer for Surgical Phase Recognition

One step surgical scene restoration for robot assisted minimally invasive surgery

Towards Robust Algorithms for Surgical Phase Recognition via Digital Twin-based Scene Representation

Surgical Video Workflow Analysis Via Visual-Language Learning

Hierarchical Semi-Supervised Learning Framework for Surgical Gesture Segmentation and Recognition Based on Multi-Modality Data

Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery

MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation

Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection

Surgical instrument recognition for instrument usage documentation and surgical video library indexing