Abstract:Objectives: Insufficient attention has been given to how information from computer-based clinical case simulations is presented, collected, and scored. Research is needed on how best to design such simulations to acquire valid performance assessment data that can act as useful feedback for educational applications. This report describes a study of a new simulation format with design features aimed at improving both its formative assessment feedback and educational function. Methods: Case simulation software (LabCAPS) was developed to target a highly focused and well-defined measurement goal with a response format that allowed objective scoring. Data from an eight-case computer-based performance assessment administered in a pilot study to 13 second-year medical students was analyzed using classical test theory and generalizability analysis. In addition, a similar analysis was conducted on an administration in a less controlled setting, but to a much large sample (n = 143), within a clinical course that utilized two random case subsets from a library of 18 cases. Results: Classical test theory case-level item analysis of the pilot assessment yielded an average case discrimination of 0.37, and all eight cases were positively discriminating (range = 0.11-0.56). Classical test theory coefficient alpha and the decision study showed the eight-case performance assessment to have an observed reliability of σ = G = 0.70. The decision study further demonstrated that a G = 0.80 could be attained with approximately 3 h and 15 min of testing. The less-controlled educational application within a large medical class produced a somewhat lower reliability for eight cases (G = 0.53). Students gave high ratings to the logic of the simulation interface, its educational value, and to the fidelity of the tasks. Conclusions: LabCAPS software shows the potential to provide formative assessment of medical students' skill at diagnostic test ordering and to provide valid feedback to learners. The perceived fidelity of the performance tasks and the statistical reliability findings support the validity of using the automated scores for formative assessment and learning. LabCAPS cases appear well designed for use as a scored assignment, for stimulating discussions in small group educational settings, for self-assessment, and for independent learning. Extension of the more highly controlled pilot assessment study with a larger sample will be needed to confirm its reliability in other assessment applications.

Enhancing Digital Simulated Laboratory Assessments: a Test of Pre-Laboratory Activities with the Learning Error and Formative Feedback Model

Digital Learning Ecosystem to Enhance Formative Assessment in Second Language Acquisition in Higher Education

Using voluntary laboratory simulations as preparatory tasks to improve conceptual knowledge and engagement

The Benefits of Participating in a Learning Assistant Program on the Metacognitive Awareness and Motivation of Learning Assistants

Effectiveness of Using a Self-Directed Learning Program to Teach Physical Examination and Health Assessment Skills: A Quasi-Experimental Study

Modified directed self-learning sessions in physiology with prereading assignments and Pecha Kucha talks: perceptions of students

A Learning Experience Design Approach to Online Professional Development for Teaching Science through the Arts: Evaluation of Teacher Content Knowledge, Self-Efficacy and STEAM Perceptions

Remote, simulation or traditional engineering teaching laboratory: a systematic literature review of assessment implementations to measure student achievement or learning

A report on the piloting of a novel computer-based medical case simulation for teaching and formative assessment of diagnostic laboratory testing

Development of Multiple Skill Laboratory Activity Model (MSLAM): An Instrument to Improve 21st Century Skills of Student

Enhancing academic performance and student engagement in health education: insights from Work Station Learning Activities (WSLA)

Augmenting the effect of virtual labs with "teacher demonstration" and "student critique" instructional designs to scaffold the development of scientific literacy

Practical Application of Assessment Theories for English Teacher through Digital Technologies

Digital performance assessment: Measure a pharmacy physics laboratory’s skill

Assessing Learner Engagement and the Impact on Academic Performance within a Virtual Learning Environment

Impacts of computer-assisted diagnostic assessment on sustainability of L2 learners’ collaborative writing improvement and their engagement modes

Digital laboratory report writing, assessment and feedback in the 21st century for an extended curriculum programme for physics

Improving Science Achievement—Is It Possible? Evaluating the Efficacy of a High School Chemistry and Physics Project-Based Learning Intervention

Learner-oriented assessment matters: testing the effects of academic buoyancy, reflective thinking, and learner enjoyment in self-assessment and test-taking anxiety management of the EFL learners

Characterizing College Science Assessments: The Three-Dimensional Learning Assessment Protocol

Assessing Digital Literacy Skills in Secondary and Higher Education: A Comprehensive Literature Survey of Validated Insttuments