Abstract:Modern operating room is becoming increasingly complex, requiring innovative intra-operative support systems. While the focus of surgical data science has largely been on video analysis, integrating surgical computer vision with language capabilities is emerging as a necessity. Our work aims to advance Visual Question Answering (VQA) in the surgical context with scene graph knowledge, addressing two main challenges in the current surgical VQA systems: removing question-condition bias in the surgical VQA dataset and incorporating scene-aware reasoning in the surgical VQA model design. First, we propose a Surgical Scene Graph-based dataset, SSG-QA, generated by employing segmentation and detection models on publicly available datasets. We build surgical scene graphs using spatial and action information of instruments and anatomies. These graphs are fed into a question engine, generating diverse QA pairs. Our SSG-QA dataset provides a more complex, diverse, geometrically grounded, unbiased, and surgical action-oriented dataset compared to existing surgical VQA datasets. We then propose SSG-QA-Net, a novel surgical VQA model incorporating a lightweight Scene-embedded Interaction Module (SIM), which integrates geometric scene knowledge in the VQA model design by employing cross-attention between the textual and the scene features. Our comprehensive analysis of the SSG-QA dataset shows that SSG-QA-Net outperforms existing methods across different question types and complexities. We highlight that the primary limitation in the current surgical VQA systems is the lack of scene knowledge to answer complex queries. We present a novel surgical VQA dataset and model and show that results can be significantly improved by incorporating geometric scene features in the VQA model design. The source code and the dataset will be made publicly available at: <a class="link-external link-https" href="https://github.com/CAMMA-public/SSG-QA" rel="external noopener nofollow">this https URL</a>

Surgical Visual Domain Adaptation: Results from the MICCAI 2020 SurgVisDom Challenge

Objective Surgical Skills Assessment and Tool Localization: Results from the MICCAI 2021 SimSurgSkill Challenge

PitVis-2023 Challenge: Workflow Recognition in videos of Endoscopic Pituitary Surgery

Adaptation of Surgical Activity Recognition Models Across Operating Rooms

Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer

Surgical Phase Recognition in Inguinal Hernia Repair—AI-Based Confirmatory Baseline and Exploration of Competitive Models

Computer vision in surgery

Dissecting self-supervised learning methods for surgical computer vision

Quantification of Robotic Surgeries with Vision-Based Deep Learning

Computer Vision in the Operating Room: Opportunities and Caveats

Computer vision in surgery: from potential to clinical value

Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery

A comprehensive survey on recent deep learning-based methods applied to surgical data

Cross-Domain Conditional Generative Adversarial Networks for Stereoscopic Hyperrealism in Surgical Training

General surgery vision transformer: A video pre-trained foundation model for general surgery

2018 Robotic Scene Segmentation Challenge

Handling Geometric Domain Shifts in Semantic Segmentation of Surgical RGB and Hyperspectral Images

Surgical data science for safe cholecystectomy: a protocol for segmentation of hepatocystic anatomy and assessment of the critical view of safety

Advancing Surgical VQA with Scene Graph Knowledge

SimCol3D -- 3D Reconstruction during Colonoscopy Challenge

Cataract-1K: Cataract Surgery Dataset for Scene Segmentation, Phase Recognition, and Irregularity Detection