Examiner workload comparison: three structured oral examination formats for the European diploma in anaesthesiology and intensive care

Mikhail Dziadzko,Andrey Varvinskiy,Rodolphe Di Loreto,Hugues Scipioni,Bazil Ateleanu,Markus Klimek,Joana Berger-Estilita
DOI: https://doi.org/10.1080/10872981.2024.2364990
IF: 6
2024-06-09
Medical Education Online
Abstract:The COVID-19 pandemic triggered transformations in academic medicine, rapidly adopting remote teaching and online assessments. Whilst virtual environments show promise in evaluating medical knowledge, their impact on examiner workload is unclear. This study explores examiner's workload during different European Diploma in Anaesthesiology and Intensive Care Part 2 Structured Oral Examinations formats. We hypothesise that online exams result in lower examiner's workload than traditional face-to-face methods. We also investigate workload structure and its correlation with examiner characteristics and marking performance. In 2023, examiner's workload for three examination formats (face-to-face, hybrid, online) using the NASA TLX instrument was prospectively evaluated. The impact of examiner demographics, candidate scoring agreement, and examination scores on workload was analysed. The overall NASA TLX score from 215 workload measurements in 142 examiners was high at 59.61 ± 14.13. The online examination had a statistically higher workload (61.65 ± 12.84) than hybrid but not face-to-face. Primary contributors to workload were mental and temporal demands, and effort. Online exams were associated with elevated frustration. Male examiners and those spending more time on exam preparation experienced a higher workload. Multiple diploma specialties and familiarity with European Diploma in Anaesthesiology and Intensive Care exams were protective against high workload. Perceived workload did not impact marking agreement or examination scores across all formats. Examiners experience high workload. Online exams are not systematically associated with decreased workload, likely due to frustration. Despite workload differences, no impact on examiner's performance or examination scores was found. The hybrid examination mode, combining face-to-face and online, was associated with a minor but statistically significant workload reduction. This hybrid approach may offer a more balanced and efficient examination process while maintaining integrity, cost savings, and increased accessibility for candidates.
education & educational research
What problem does this paper attempt to address?
The paper primarily explores the impact of different examination formats on examiners' workload in the structured oral examination of the European Diploma in Anaesthesiology and Intensive Care (EDAIC) Part 2. The research background is set during the COVID-19 pandemic, where medical education rapidly shifted to remote teaching and online assessments. Although virtual environments have shown potential in assessing medical knowledge, their impact on examiners' workload remains unclear. Therefore, the researchers investigated examiners' workload by comparing three different examination formats—Face-to-Face (F2F), hybrid mode, and fully online. The study hypothesized that online examinations would reduce examiners' workload compared to traditional face-to-face examinations. Additionally, the study examined the structure of the workload and its correlation with examiners' characteristics and scoring performance. The main findings of the study include: - The overall workload of examiners was high (mean NASA TLX score of 59.61 ± 14.13), with the workload for online examinations being significantly higher than for the hybrid mode but not significantly different from face-to-face examinations. - The main contributing factors to the workload included mental demand, time demand, and effort. - Online examinations were associated with higher frustration levels. - Male examiners and those who spent more time preparing for the examination experienced higher workloads. - Examiners with multiple professional qualifications or familiarity with the EDAIC examination were better able to cope with high workloads. - Despite the differences in workload, there was no impact on the consistency of examiners' scoring and examination results. - The hybrid examination mode was associated with a slight but statistically significant reduction in workload, which may provide a more balanced and efficient examination process. The conclusion points out that structured oral examinations for EDAIC Part 2 consistently involve a high workload. While the hybrid examination mode does not significantly alleviate the high workload associated with online examinations, it offers a potential method to balance efficiency, fairness, and accessibility. Despite the high workload, there was no significant impact on the quality of scoring, reflecting the professional commitment of the examiners.