Abstract:Computer-assisted surgery (CAS) aims to provide the surgeon with the right type of assistance at the right moment. Such assistance systems are especially relevant in laparoscopic surgery, where CAS can alleviate some of the drawbacks that surgeons incur. For many assistance functions, e.g. displaying the location of a tumor at the appropriate time or suggesting what instruments to prepare next, analyzing the surgical workflow is a prerequisite. Since laparoscopic interventions are performed via endoscope, the video signal is an obvious sensor modality to rely on for workflow analysis. Image-based workflow analysis tasks in laparoscopy, such as phase recognition, skill assessment, video indexing or automatic annotation, require a temporal distinction between video frames. Generally computer vision based methods that generalize from previously seen data are used. For training such methods, large amounts of annotated data are necessary. Annotating surgical data requires expert knowledge, therefore collecting a sufficient amount of data is difficult, time-consuming and not always feasible. In this paper, we address this problem by presenting an unsupervised method for training a convolutional neural network (CNN) to differentiate between laparoscopic video frames on a temporal basis. We extract video frames at regular intervals from 324 unlabeled laparoscopic interventions, resulting in a dataset of approximately 2.2 million images. From this dataset, we extract image pairs from the same video and train a CNN to determine their temporal order. To solve this problem, the CNN has to extract features that are relevant for comprehending laparoscopic workflow. Furthermore, we demonstrate that such a CNN can be adapted for surgical workflow segmentation. We performed image-based workflow segmentation on a publicly available dataset of 7 cholecystectomies and 9 colorectal interventions.

Temporal-based Swin Transformer network for workflow recognition of surgical video

An Empirical Study on Activity Recognition in Long Surgical Videos

Convolutional-de-convolutional neural networks for recognition of surgical workflow

SF-TMN: SlowFast temporal modeling network for surgical phase recognition

Aggregating Long-Term Context for Learning Laparoscopic and Robot-Assisted Surgical Workflows

Video-SwinUNet: Spatio-temporal Deep Learning Framework for VFSS Instance Segmentation

Swin-Tempo: Temporal-Aware Lung Nodule Detection in CT Scans as Video Sequences Using Swin Transformer-Enhanced UNet

Surgformer: Surgical Transformer with Hierarchical Temporal Attention for Surgical Phase Recognition

Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentation

Cascade Multi-Level Transformer Network for Surgical Workflow Analysis

Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images

Unsupervised temporal context learning using convolutional neural networks for laparoscopic workflow analysis

Hierarchical Semi-Supervised Learning Framework for Surgical Gesture Segmentation and Recognition Based on Multi-Modality Data

Friends Across Time: Multi-Scale Action Segmentation Transformer for Surgical Phase Recognition

Surgical Phase Recognition of Short Video Shots Based on Temporal Modeling of Deep Features

TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks

Frequency-Based Temporal Analysis Network for Accurate Phase Recognition from Surgical Videos

Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition

Multimodal semi-supervised learning for online recognition of multi-granularity surgical workflows

SSTrans-Net: Smart Swin Transformer Network for medical image segmentation

Endo3d: Online Workflow Analysis For Endoscopic Surgeries Based On 3d Cnn And Lstm