Abstract:Surgical workflow analysis is essential to help optimize surgery by encouraging efficient communication and the use of resources. However, the performance of phase recognition is limited by the use of information related to the presence of surgical instruments. To address the problem, we propose visual modality-based multimodal fusion for surgical phase recognition to overcome the limited diversity of information such as the presence of instruments. Using the proposed methods, we extracted a visual kinematics-based index related to using instruments, such as movement and their interrelations during surgery. In addition, we improved recognition performance using an effective convolutional neural network (CNN)-based fusion method for visual features and a visual kinematics-based index (VKI). The visual kinematics-based index improves the understanding of a surgical procedure since information is related to instrument interaction. Furthermore, these indices can be extracted in any environment, such as laparoscopic surgery, and help obtain complementary information for system kinematics log errors. The proposed methodology was applied to two multimodal datasets, a virtual reality (VR) simulator-based dataset (PETRAW) and a private distal gastrectomy surgery dataset, to verify that it can help improve recognition performance in clinical environments. We also explored the influence of a visual kinematics-based index to recognize each surgical workflow by the instrument's existence and the instrument's trajectory. Through the experimental results of a distal gastrectomy video dataset, we validated the effectiveness of our proposed fusion approach in surgical phase recognition. The relatively simple yet index-incorporated fusion we propose can yield significant performance improvements over only CNN-based training and exhibits effective training results compared to fusion based on Transformers, which require a large amount of pre-trained data.

Multimodal data fusion framework enhanced robot-assisted minimally invasive surgery

Human Activity Recognition Enhanced Robot-Assisted Minimally Invasive Surgery

On the Fly Robotic-Assisted Medical Instrument Planning and Execution Using Mixed Reality

An Automated Skill Assessment Framework Based on Visual Motion Signals and a Deep Neural Network in Robot-Assisted Minimally Invasive Surgery

Learning-Based Multimodal Information Fusion and Behavior Recognition of Vascular Interventionists' Operating Skills

Multi-Modal Autonomous Ultrasound Scanning for Efficient Human–Machine Fusion Interaction

Multimodal Sensors and ML‐Based Data Fusion for Advanced Robots

Collaborative Framework for Robot-Assisted Minimally Invasive Surgery Using a 7-Dof Anthropomorphic Robot

Multimodal information fusion for human-robot interaction

Multi-Modal Data Fusion in Enhancing Human-Machine Interaction for Robotic Applications: A Survey

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Adaptive sensor fusion labeling framework for hand pose recognition in robot teleoperation

Robotic Teleoperation Systems Using a Wearable Multimodal Fusion Device

Multi-sensor fusion for body sensor network in medical human–robot interaction scenario

An adaptive reinforcement learning-based multimodal data fusion framework for human-robot confrontation gaming

Progress and Prospects of Multimodal Fusion Methods in Physical Human–Robot Interaction: A Review

A Multilayer and Multimodal-Fusion Architecture for Simultaneous Recognition of Endovascular Manipulations and Assessment of Technical Skills

Multimodal fusion and human-robot interaction control of an intelligent robot

MEDiC: Autonomous Surgical Robotic Assistance to Maximizing Exposure for Dissection and Cautery

Sensor Fusion-Based Anthropomorphic Control of a Robotic Arm

Visual modalities-based multimodal fusion for surgical phase recognition