Abstract:Objectives: Insufficient attention has been given to how information from computer-based clinical case simulations is presented, collected, and scored. Research is needed on how best to design such simulations to acquire valid performance assessment data that can act as useful feedback for educational applications. This report describes a study of a new simulation format with design features aimed at improving both its formative assessment feedback and educational function. Methods: Case simulation software (LabCAPS) was developed to target a highly focused and well-defined measurement goal with a response format that allowed objective scoring. Data from an eight-case computer-based performance assessment administered in a pilot study to 13 second-year medical students was analyzed using classical test theory and generalizability analysis. In addition, a similar analysis was conducted on an administration in a less controlled setting, but to a much large sample (n = 143), within a clinical course that utilized two random case subsets from a library of 18 cases. Results: Classical test theory case-level item analysis of the pilot assessment yielded an average case discrimination of 0.37, and all eight cases were positively discriminating (range = 0.11-0.56). Classical test theory coefficient alpha and the decision study showed the eight-case performance assessment to have an observed reliability of σ = G = 0.70. The decision study further demonstrated that a G = 0.80 could be attained with approximately 3 h and 15 min of testing. The less-controlled educational application within a large medical class produced a somewhat lower reliability for eight cases (G = 0.53). Students gave high ratings to the logic of the simulation interface, its educational value, and to the fidelity of the tasks. Conclusions: LabCAPS software shows the potential to provide formative assessment of medical students' skill at diagnostic test ordering and to provide valid feedback to learners. The perceived fidelity of the performance tasks and the statistical reliability findings support the validity of using the automated scores for formative assessment and learning. LabCAPS cases appear well designed for use as a scored assignment, for stimulating discussions in small group educational settings, for self-assessment, and for independent learning. Extension of the more highly controlled pilot assessment study with a larger sample will be needed to confirm its reliability in other assessment applications.

SCRAPS: Introducing a Student-Centered Resident-Administered PACS Simulator for Medical Student Radiology Education

Digital hands-on learning in radiology-design and evaluation of a PACS-based concept for medical students

The Time Has Come: a Paradigm Shift in Diagnostic Radiology Education via Simulation Training

Interactive teaching environment for diagnostic radiography with real-time X-ray simulation and patient positioning

Development and Assessment of Simulation-Based Point-of-Care Ultrasound Curriculum in Undergraduate Medical Education

A report on the piloting of a novel computer-based medical case simulation for teaching and formative assessment of diagnostic laboratory testing

Efficiency of Simulation-Based Learning Using an ABC POCUS Protocol on a High-Fidelity Simulator

A simulation-based acute care curriculum for pediatric emergency medicine fellowship training programs

760 Simulation Enhances Resident Performance Using Autologous Skin Cell Spray

Improving medical students' responses to emergencies with a simulated cross-cover paging curriculum

Assessment of Medical Students' Ability to Integrate Point-of-Care Cardiac Ultrasound Into a Case-Based Simulation After a Short Intervention

Simulation‐Based Education for Urgent Medical Complications Common to the Rehabilitation Setting: An Educational Program for Physical Medicine and Rehabilitation Residents

Simulated medication administration for vulnerable populations using scanning technology: a quasi-experimental pilot study

Dissemination of a Pediatric Musculoskeletal POCUS Scoring System via Virtual Education: A Proof-of-Concept Study

Dissemination, Utilization, and Satisfaction With Emergency Medicine Resident Simulation Curriculum for Pediatrics (EM ReSCu Peds) by North American Residency Programs

Changing learning with new interactive and media-rich instruction environments: virtual labs case study report

Implementation of Medical Students as Radiology Reading Room Coordinators

Creation of novel pediatric academic curriculum and its evaluation using mixed methods

Using modified information delivery to enhance the traditional pharmacy OSCE program at TMU - a pilot study

Sonographer Educator in the Emergency Department to Prevent POCUS Skills Attrition: A Novel Education Intervention

The Electronic Health Record Objective Structured Clinical Examination Station: Assessing Student Competency in Patient Notes and Patient Interaction