Evaluating the Impact on Clinical Task Efficiency of a Natural Language Processing Algorithm for Searching Medical Documents: Prospective Crossover Study
Eunsoo H Park,Hannah I Watson,Felicity V Mehendale,Alison Q O'Neil,Clinical Evaluators,Fiona Auld,Anna Barton,Rong Bing,Marcus Boyd,Cameron Brown,Khai Syuen Chew,Jane Yi Chiam,Vanessa Chou,Luisa Ciriello,George Cooper,Iona Cutworth,Elizabeth Daly,Jamie Donachie,Greta Economides,Vivienne Evans,Magdalena Gabrysiak,Eilidh Gunn,Mohamed Hamed,Hamzah Hanif,Ewen Harrison,Kylla Hernandez,Lana Huang,Katie Hunter,Haider Khan,David Kluth,Niki Kouvroukoglou,Barbora Krivankova,Tommy Le,Charles Leeson-Payne,Alinah Sum-Ping Leung,Keziah Lewis,Jenny Lockhart,Jack Lugue,Angus MacLeod,Tomos Morgon,Ellen Murgitroyd,Sarah Murphy,Abhishek Nambiar,Sumrah Naqvi,Helen O'Neil,Yusuke Onishi,Lisa Ragunathan,Nikita Rana,Risako Sakatsume,Qi Shun Yong,Faye Sikora,Lucy Taylor,Evangelos Tzolos,Miriam Veenhuizen,Philipa Veenhuizen,Emma Warburton,Olivia Yu,Sydney Zides
DOI: https://doi.org/10.2196/39616
2022-10-26
Abstract:Background: Information retrieval (IR) from the free text within electronic health records (EHRs) is time consuming and complex. We hypothesize that natural language processing (NLP)-enhanced search functionality for EHRs can make clinical workflows more efficient and reduce cognitive load for clinicians. Objective: This study aimed to evaluate the efficacy of 3 levels of search functionality (no search, string search, and NLP-enhanced search) in supporting IR for clinical users from the free text of EHR documents in a simulated clinical environment. Methods: A clinical environment was simulated by uploading 3 sets of patient notes into an EHR research software application and presenting these alongside 3 corresponding IR tasks. Tasks contained a mixture of multiple-choice and free-text questions. A prospective crossover study design was used, for which 3 groups of evaluators were recruited, which comprised doctors (n=19) and medical students (n=16). Evaluators performed the 3 tasks using each of the search functionalities in an order in accordance with their randomly assigned group. The speed and accuracy of task completion were measured and analyzed, and user perceptions of NLP-enhanced search were reviewed in a feedback survey. Results: NLP-enhanced search facilitated more accurate task completion than both string search (5.14%; P=.02) and no search (5.13%; P=.08). NLP-enhanced search and string search facilitated similar task speeds, both showing an increase in speed compared to the no search function, by 11.5% (P=.008) and 16.0% (P=.007) respectively. Overall, 93% of evaluators agreed that NLP-enhanced search would make clinical workflows more efficient than string search, with qualitative feedback reporting that NLP-enhanced search reduced cognitive load. Conclusions: To the best of our knowledge, this study is the largest evaluation to date of different search functionalities for supporting target clinical users in realistic clinical workflows, with a 3-way prospective crossover study design. NLP-enhanced search improved both accuracy and speed of clinical EHR IR tasks compared to browsing clinical notes without search. NLP-enhanced search improved accuracy and reduced the number of searches required for clinical EHR IR tasks compared to direct search term matching.