Abstract:Background context: The majority of surgical training is conducted in real-world operations. High-fidelity surgical simulators may provide a safer environment for surgical training. However, the extent that it reflects real-world operations and surgical ability is often poorly characterized. Purpose: (1) Assess the validity and fidelity of a surgical simulator; (2) Examine the quantitative relationship between simulation performance and markers of real-world ability; (3) Establish thresholds for surgical expertise, and estimate their external validity and accuracy. Study design/setting: A cohort study of surgeons at a British neurosurgical center. Study sample: 10 early-career "novice" surgeons and 8 board-certified "expert" neurosurgeons. Outcomes measures: (1) Face and content validity, and visual and haptic fidelity; (2) Construct validity; (3) Predictive and discriminative utility of quantitative performance thresholds. Methods: Participants performed unilateral lumbar decompressions on high-fidelity spinal simulators that replicate the bony and soft tissue anatomy along with physiological processes such as bleeding and CSF leaks. Operating times, measured from first surgical action to either self-perceived satisfactory decompression or the end of allocated time, were recorded. The performance was also assessed independently by two blinded spinal subspecialist neurosurgeons using OSATS, a validated surgical assessment tool that utilizes five-point scales on a variety of technical domains to grade the overall technical proficiency. Validity and fidelity were assessed by expert neurosurgeons using quantitative questionnaires. Construct validity was assessed by ordinal regression of simulation performance against real-world surgical grade and portfolio. Thresholds of expert status by simulation performance was established, and their predictive and discriminative utility assessed by crossvalidation accuracy and AUC-ROC. Results: Operating time and expert assessments of simulation performance (OSATS) were strong and significant predictors of surrogate markers of real-world surgical ability. The thresholds for expert status were operating time of 15 minutes and modified OSATS score of 15/20. These thresholds predicted expert status with 84.2% and 71.4% accuracy respectively. Strong discriminative ability was demonstrated by AUC-ROC of 0.95 and 0.83 respectively. All expert surgeons agreed that RealSpine simulators demonstrate high face validity, and high visual and haptic fidelity, with overall scores showing statistically significant agreement on these items (all scores at least 4/5, P<.0001). There was less consensus on content validity, but with still significant overall agreement (average score: 3.75/5, P=0.023). Conclusions: Real-world surgical ability and experience can be accurately predicted by defining objective quantitative thresholds on high-fidelity simulations. The thresholds established here, along with other data presented in this paper, may inform objectives and standards to be established in a spinal surgical training curriculum.

Development and validation of an objective virtual reality tool for assessing technical aptitude among potential candidates for surgical training

A novel augmented reality simulator for skills assessment in minimal invasive surgery

Validation of a novel virtual reality robotic simulator

Systematic approach for content and construct validation: Case studies for arthroscopy and laparoscopy

Evaluation of a virtual reality simulator for arthroscopy skills development

Construct Validity of a Virtual Reality Simulator for Surgical Training in Knee Arthroscopy

Development and Validation of a Virtual Reality Simulator for Robot-Assisted Minimally Invasive Liver Surgery Training

Gaze behavior is related to objective technical skills assessment during virtual reality simulator-based surgical training: a proof of concept

Evaluation of skill level between trainees and community orthopaedic surgeons using a virtual reality arthroscopic knee simulator

Use of Virtual Reality in the Education of Orthopaedic Procedures: A Randomised Control Study in Early Validation of a Novel Virtual Reality Simulator

Usefulness of virtual reality in assessment of medical student laparoscopic skill

VR Isle Academy: A VR Digital Twin Approach for Robotic Surgical Skill Development

Distribution of innate psychomotor skills recognized as important for surgical specialization in unconditioned medical undergraduates

Virtual reality informs clinical observation tool

Design and evaluation of UltRASim: An immersive simulator for learning ultrasound-guided regional anesthesia basic skills

Validation of a surgical simulator and establishment of quantitative performance thresholds - RealSpine simulation system for open lumbar decompressions

Implementation of a Virtual Reality Based Digital-Twin Robotic Minimally Invasive Surgery Simulator

Training validity of laparoscopic virtual reality simulator

Validity of an immersive virtual reality training system for orthognathic surgical education

Proficiency Levels and Validity Evidence for Scoring Metrics for a Virtual Reality and Inanimate Robotic Surgery Simulation Curriculum

Validation of a novel, low-fidelity virtual reality simulator and an artificial intelligence assessment approach for peg transfer laparoscopic training