A pilot study of the use of artificial intelligence with high‐fidelity simulations in assessing endovascular procedural competence independent of a human examiner

Erin C. Saricilar,Annette Burgess,Anthony Freeman

DOI: https://doi.org/10.1111/ans.18484

IF: 1.7

2023-04-25

ANZ Journal of Surgery

Abstract:With increase endovascular training in vascular surgery, assessment methods need to shift to capture these nuanced skills. A high‐fidelity simulator was set up to capture raw metric data and compared to a standardized assessment method. There were significant correlations between some metric scores and an artificial intelligence model had moderate correlation with these score. Background With increased need for vascular surgery trainees to gain endovascular surgery proficiency, current models of case‐numbers and subjective visual assessment are inadequate in capturing the skills required in endovascular surgery. We explored the use of high‐fidelity simulators in (1) assessing endovascular surgical competence; (2) clinical decision making; and (3) the reliability of an artificial intelligence (AI) assessor. Methods Registrars, fellows and consultants from vascular surgery, interventional radiology and general surgery performed identical procedures on a high‐fidelity simulator. Performance was independently assessed using a modified Reznick scale. Scores were compared to raw metric data extracted from the simulator, objective scores extracted from the recordings and analysed by AI. Results 22 participants were enrolled from vascular surgery (n = 6, 27.3%), interventional radiology (n = 10, 45.5%) and general surgery (n = 6, 27.3%). There were 12 trainees, 2 fellows and 8 consultants. Significant correlations between raw metric data and all categories of the modified Reznick scale except 'respect for tissue' were found. An AI demonstrated positive reliability in all categories, with some predictions being moderately correlated. Conclusion The use of high‐fidelity simulators to assess endovascular surgical competence has comparable correlations to the traditional assessment methods with global rating scales, which can be used in formative assessment. AI demonstrates an ability to support assessment but requires further research.

surgery