Abstract:Objectives: Insufficient attention has been given to how information from computer-based clinical case simulations is presented, collected, and scored. Research is needed on how best to design such simulations to acquire valid performance assessment data that can act as useful feedback for educational applications. This report describes a study of a new simulation format with design features aimed at improving both its formative assessment feedback and educational function. Methods: Case simulation software (LabCAPS) was developed to target a highly focused and well-defined measurement goal with a response format that allowed objective scoring. Data from an eight-case computer-based performance assessment administered in a pilot study to 13 second-year medical students was analyzed using classical test theory and generalizability analysis. In addition, a similar analysis was conducted on an administration in a less controlled setting, but to a much large sample (n = 143), within a clinical course that utilized two random case subsets from a library of 18 cases. Results: Classical test theory case-level item analysis of the pilot assessment yielded an average case discrimination of 0.37, and all eight cases were positively discriminating (range = 0.11-0.56). Classical test theory coefficient alpha and the decision study showed the eight-case performance assessment to have an observed reliability of σ = G = 0.70. The decision study further demonstrated that a G = 0.80 could be attained with approximately 3 h and 15 min of testing. The less-controlled educational application within a large medical class produced a somewhat lower reliability for eight cases (G = 0.53). Students gave high ratings to the logic of the simulation interface, its educational value, and to the fidelity of the tasks. Conclusions: LabCAPS software shows the potential to provide formative assessment of medical students' skill at diagnostic test ordering and to provide valid feedback to learners. The perceived fidelity of the performance tasks and the statistical reliability findings support the validity of using the automated scores for formative assessment and learning. LabCAPS cases appear well designed for use as a scored assignment, for stimulating discussions in small group educational settings, for self-assessment, and for independent learning. Extension of the more highly controlled pilot assessment study with a larger sample will be needed to confirm its reliability in other assessment applications.

Computer-based Simulation Training to Improve Learning Outcomes in Mannequin-based Simulation Exercises

Learning curves and long-term outcome of simulation-based thoracentesis training for medical students

Standardized Patients Versus Mannequins in Mental Health Simulation

Retention of Advanced Cardiac Life Support Knowledge and Skills Following High-Fidelity Mannequin Simulation Training

Hybrid Simulation Experience-Hybrid Simulator Model vs. Manikin in Bladder Catheterization Procedure: A Pilot Study

Comparison of Mannequin-Based Simulation Training Method with Virtual Training Method on Nursing Students' Learning Cardiopulmonary Resuscitation: A Controlled Randomized Parallel Trial

Comparing virtual reality and simulation to teach the assessment and management of acute surgical scenarios: A pilot study

Impact of Simulation Training on Core Skill Competency of Undergraduate Medical Students

Effectiveness of applying clinical simulation scenarios and integrating information technology in medical-surgical nursing and critical nursing courses

Plastic with personality: Increasing student engagement with manikins

Evaluating a computer-based experiential learning simulation: a case study using criterion-referenced testing

A Simulation to Improve Understanding and Communication of Ethical Dilemmas That Surround Brain Death

Does simulation lab increase confidence in Occupational Therapy Students' clinical skills: A pilot study

Exploring the Merit of Simulation-Based Education: A Systematic Review and Meta-Analysis

Telepresent mechanical ventilation training versus traditional instruction: a simulation-based pilot study

Subjective learning gain from a simulation-based health management course: a mixed methods study

A report on the piloting of a novel computer-based medical case simulation for teaching and formative assessment of diagnostic laboratory testing

Gross Anatomy Teaching for Medical Undergraduates Through Computer-Based Simulation: Introduction and Evaluation of Effectiveness

Evaluating the effects of comprehensive simulation on central venous catheterization training: a comparative observational study

Use of High-Fidelity Simulation as an Adjunct to Basic Life Support Training To Promote Team-Based Resuscitation Skills: A Mixed-Methods Pilot Study

Effectiveness, merits and challenges of simulation-based online clinical skills teaching compared to face-to-face teaching – a case–control study