Abstract:Background context: The majority of surgical training is conducted in real-world operations. High-fidelity surgical simulators may provide a safer environment for surgical training. However, the extent that it reflects real-world operations and surgical ability is often poorly characterized. Purpose: (1) Assess the validity and fidelity of a surgical simulator; (2) Examine the quantitative relationship between simulation performance and markers of real-world ability; (3) Establish thresholds for surgical expertise, and estimate their external validity and accuracy. Study design/setting: A cohort study of surgeons at a British neurosurgical center. Study sample: 10 early-career "novice" surgeons and 8 board-certified "expert" neurosurgeons. Outcomes measures: (1) Face and content validity, and visual and haptic fidelity; (2) Construct validity; (3) Predictive and discriminative utility of quantitative performance thresholds. Methods: Participants performed unilateral lumbar decompressions on high-fidelity spinal simulators that replicate the bony and soft tissue anatomy along with physiological processes such as bleeding and CSF leaks. Operating times, measured from first surgical action to either self-perceived satisfactory decompression or the end of allocated time, were recorded. The performance was also assessed independently by two blinded spinal subspecialist neurosurgeons using OSATS, a validated surgical assessment tool that utilizes five-point scales on a variety of technical domains to grade the overall technical proficiency. Validity and fidelity were assessed by expert neurosurgeons using quantitative questionnaires. Construct validity was assessed by ordinal regression of simulation performance against real-world surgical grade and portfolio. Thresholds of expert status by simulation performance was established, and their predictive and discriminative utility assessed by crossvalidation accuracy and AUC-ROC. Results: Operating time and expert assessments of simulation performance (OSATS) were strong and significant predictors of surrogate markers of real-world surgical ability. The thresholds for expert status were operating time of 15 minutes and modified OSATS score of 15/20. These thresholds predicted expert status with 84.2% and 71.4% accuracy respectively. Strong discriminative ability was demonstrated by AUC-ROC of 0.95 and 0.83 respectively. All expert surgeons agreed that RealSpine simulators demonstrate high face validity, and high visual and haptic fidelity, with overall scores showing statistically significant agreement on these items (all scores at least 4/5, P<.0001). There was less consensus on content validity, but with still significant overall agreement (average score: 3.75/5, P=0.023). Conclusions: Real-world surgical ability and experience can be accurately predicted by defining objective quantitative thresholds on high-fidelity simulations. The thresholds established here, along with other data presented in this paper, may inform objectives and standards to be established in a spinal surgical training curriculum.

Validity of task-specific metrics for assessment in perineal proctectomy

Validity and reliability evidence support task-specific metrics for laparoscopic fundoplication

Proficiency Levels and Validity Evidence for Scoring Metrics for a Virtual Reality and Inanimate Robotic Surgery Simulation Curriculum

Validation of a novel virtual reality robotic simulator

Validation of a surgical simulator and establishment of quantitative performance thresholds - RealSpine simulation system for open lumbar decompressions

Systematic approach for content and construct validation: Case studies for arthroscopy and laparoscopy

Development and preliminary validation of a new task-based objective procedure-specific assessment of inguinal hernia repair procedural safety

Creation and validation of a novel low-cost dry lab for early resident training and assessment of robotic prostatectomy technical proficiency

Objective and Automated Quantification of Instrument Handling for Open Surgical Suturing Skill Assessment: A Simulation-Based Study

Development and validation of surgical training tool: cystectomy assessment and surgical evaluation (CASE) for robot-assisted radical cystectomy for men

Scoring metrics for assessing skills in arthroscopic rotator cuff repair: performance comparison study of novice and expert surgeons

Design and validation of an assessment tool for open surgical procedures

Initial Validation of a Virtual-Reality Learning Environment for Prostate Biopsies: Realism Matters!

Development and validation of 3-dimensional simulators for penile prosthesis surgery

Can we become better robot surgeons through simulator practice?

What are clinically relevant performance metrics in robotic surgery? A systematic review of the literature

Qualitative and Quantitative Assessment of Invasive Surgery Skill Based on a Custom Training Simulator

Development and validation of an objective virtual reality tool for assessing technical aptitude among potential candidates for surgical training

Validation of a Simulation Model for Robotic Myomectomy

Face and Content Validation of the Virtual Reality Transurethral Prostatic Resection Simulator

Development and validation of a simulation training platform for the ligation of deep dorsal vein complex in radical prostatectomy