Abstract:Introduction A team of biomedical engineers, neurosurgeons, and a medical educator, created and evaluated preliminary validity evidence of a high-fidelity simulator used to train ventriculostomy skills. Description A scale reproduction of an adult skull and relevant anatomy was created. Twelve neurosurgery residents, one fellow, and five attendings (n=17) from three academic medical centers performed the simulated creation of a ventricolostomy. Participants rated the simulator using a 37-item survey across five domains. Quality of the simulator was scored from “Not at all realistic” (1) to “Highly realistic” (4). Self-reported ability to perform relevant tasks was rated from “Too difficult to perform (0) to “Too easy to perform” (4). Validity evidence relevant to test content was evaluated using a Rasch model, while evidence relevant to internal structure (inter-item reliability, and inter-rater agreement) was evaluated using traditional methods. Conclusion Analyses indicated attendings had statistically higher ratings (M=3.4/4.0) than fellows (M=3.3), and residents (M=3.0), p=.02. Domain means were 3.9 (Value), 3.5 (Physical attributes), 3.4 (Realism of experience), 3.3 (Relevance), and 2.9 (Ability). Inter-item consistency across domains were estimated to be moderate-high (.65, .92), and inter-rater agreement regarding simulator characteristics was high [ICC(2,k)= .90]. Rating differences were found across institutions, p=.001, and are reviewed. The observed global rating (2.5) indicated the simulator can be considered for teaching ventriculostomies, but could be improved slightly. Ratings indicated the simulator was valuable as a learning tool, but could be improved with minor modifications. The most commonly-suggested improvement was the addition of ears as an added anatomical landmark. References 1. Haji FA, Dubrowski A, Drake J, de Ribaupierre S. Needs assessment for simulation training in neuroendoscopy: a Canadian national survey. J Neurosurg. 2013 Feb;118(2):250-7. doi: 10.3171/2012.10.JNS12767. Epub 2012 Dec 7 2. Schirmer CM, Elder JB, Roitberg B, Lobel DA. Virtual reality-based simulation training for ventriculostomy: an evidence-based approach. Neurosurgery. 2013 Oct;73 Suppl 1:66-73. doi: 10.1227/NEU.0000000000000074. 3. Choudhury N, Gélinas-Phaneuf N, Delorme S, Del Maestro R. Fundamentals of neurosurgery: virtual reality tasks for training and evaluation of technical skills. World Neurosurg. 2013 Nov;80(5):e9-19. doi: 10.1016/j.wneu.2012.08.022. Epub 2012 Nov 23. 4. Korndorffer JR Jr, Kasten SJ, Downing SM. A call for the utilization of consensus standards in the surgical education literature. Am J Surg. 2010 Jan;199(1):99-104. doi: 10.1016/j.amjsurg.2009.08.018 5. Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R. Technology-enhanced simulation to assess health professionals: A systematic review of validity evidence, research methods, and reporting quality. Acad Med. 2013 Jun;88(6):872-83. doi: 10.1097/ACM.0b013e31828ffdcf. 6. Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract. 2014 May;19(2):233-50. doi: 10.1007/s10459-013-9458-4. Epub 2013 May 2. 7. Standards for Educational and Psychological Testing, 1999, American Educational Research Association, American Psychological Association and National Council on Measurement in Education: American Educational Research. 8. Rasch G. (1960/1980). Probabilistic models for some intelligence and attainment tests. (Copenhagen, Danish Institute for Educational Research), expanded edition (1980) with foreword and afterword by BD Wright. Chicago, IL: The University of Chicago Press. 9. Hamilton JM, et al., Toward effective pediatric minimally invasive surgical simulation. J Pediatr Surg, 2011. 46(1): p. 138-44. Disclosures None

Using an application to measure trainees’ procedural knowledge before chest tube insertion

Learning curves and long-term outcome of simulation-based thoracentesis training for medical students

A structured simulation‐based mastery learning curriculum in chest tube insertion results in superior skills compared to traditional training programs

Validation of a Novel Mobile Application for Assessing Pediatric Tracheostomy Emergency Simulations

Using Artificial Intelligence to Gauge Competency on a Novel Laparoscopic Training System

Face and Content Validity of a Probe Tube Placement Training Simulator

Collating evidence to support the validation of a simulated laparotomy incision and closure-training model

Development and validation of a simulation-based assessment of operative competence for higher specialist trainees in general surgery

An observer tool to enhance learning of incoming anesthesia residents' skills during simulation training of central venous catheter insertion: a randomized controlled trial

The use of objective assessments in the evaluation of technical skills in cardiothoracic surgery: a systematic review

Board #322 - Technology Innovation Development and Preliminary Validation of a Novel Ventriculostomy Simulator (Submission #8635)

[Assessment tool's validation study for a simulation-based skin suturing skill]

Multi-institutional beta testing of a novel cervical esophagogastric anastomosis simulator

Proficiency Levels and Validity Evidence for Scoring Metrics for a Virtual Reality and Inanimate Robotic Surgery Simulation Curriculum

Comparison of multimodal active learning and single-modality procedural simulation for central venous catheter insertion for incoming residents in anesthesiology: a prospective and randomized study

Validation of a virtual intracorporeal suturing simulator

Evaluating the effects of comprehensive simulation on central venous catheterization training: a comparative observational study

Gathering Validity Evidence for a Simulation-Based Test of Otoscopy Skills

Endotracheal Intubation Performance in Novices: Comparing the Training Effect Resulting From Two Airway Management Training Simulators

A Simulated Tracheostomy Tube Change Educational Intervention to Promote Competency Among Novice Healthcare Professionals: A Repeated Measures Study

Validity and reliability evidence support task-specific metrics for laparoscopic fundoplication