Managing Risk and Quality of AI in Healthcare: Are Hospitals Ready for Implementation?
Arian Ranjbar,Eilin Wermundsen Mork,Jesper Ravn,Helga Brøgger,Per Myrseth,Hans Peter Østrem,Harry Hallock
DOI: https://doi.org/10.2147/rmhp.s452337
2024-04-10
Risk Management and Healthcare Policy
Abstract:Arian Ranjbar, 1 Eilin Wermundsen Mork, 1 Jesper Ravn, 1 Helga Brøgger, 2 Per Myrseth, 2 Hans Peter Østrem, 3 Harry Hallock 2 1 Medical Technology and E-Health, Akershus University Hospital, Lørenskog, Norway; 2 Group Research and Development, DNV AS, Høvik, Norway; 3 Business Assurance, DNV AS, Høvik, Norway Correspondence: Arian Ranjbar, Akershus University Hospital, Sykehusveien 25, 1478, Lørenskog, Norway, Tel +46700436768, Email Artificial intelligence (AI) provides a unique opportunity to help meet the demands of the future healthcare system. However, hospitals may not be well equipped to handle safe and effective development and/or procurement of AI systems. Furthermore, upcoming regulations such as the EU AI Act may enforce the need to establish new management systems, quality assurance and control mechanisms, novel to healthcare organizations. This paper discusses challenges in AI implementation, particularly potential gaps in current management systems (MS), by reviewing the harmonized standard for AI MS, ISO 42001, as part of a gap analysis of a tertiary acute hospital with ongoing AI activities. Examination of the industry agnostic ISO 42001 reveals a technical debt within healthcare, aligning with previous research on digitalization and AI implementation. To successfully implement AI with quality assurance in mind, emphasis should be put on the foundation and structure of the healthcare organizations, including both workforce and data infrastructure. Keywords: artificial intelligence, management systems, quality assurance, risk management, implementation The rapid advancements in Artificial Intelligence (AI) have significantly influenced several domains. As the technology continues to mature, attention has been drawn to implementing AI into healthcare, with the potential of improving effectiveness, personalizing treatment and diagnostics, improving patient safety, meeting the rising healthcare demands caused by an ageing population, and improving homecare to relieve the burden on often understaffed hospitals. 1 However, AI differs significantly from conventional software, thus, to mitigate the new associated risks, AI requires new methods for development, procurement, implementation, and management. Given the risk that AI poses in healthcare and other sectors, entities such as the WHO have proposed regulatory considerations and frameworks for control and regulation. 2 In the US this is being achieved partly by the "Blueprint for AI Bill", but as it is non-binding, it requires implementation by all the federal agencies. 3 Conversely, in the EU this is being addressed by the centralized EU AI Act. 4,5 Globally, standards on AI are being developed by multiple organizations such as ISO/IEC JTC 1/SC 42, one being ISO/IEC 42001 Information technology – Artificial intelligence – Management system, and this has also been proposed as one of numerous future harmonized standards for the EU AI Act. 6 Specifically, ISO 42001 provides guidance for establishing, implementing, maintaining and continually improving an AI management system within the context of an organization. The introduction of ISO 42001 provides a standardized framework for management of AI and the corresponding risks. This paper adds to the discussion of potential challenges of AI implementation and operation management within healthcare, by reviewing the new standard as part of a gap analysis conducted at Akershus University Hospital. AI as a field has a long history, and with its development the scope and definition of terminology, including subfields such as machine learning, have been dynamic. 7 For the purpose of this paper we consider AI-systems in alignment with the ISO-definition: "the engineered system that generates outputs such as content, forecasts, recommendations or decisions for a given set of human-defined objectives". 8 Following the rapid development during the last decade, implementation of AI has become an increasingly investigated field, with an extensive literature on the corresponding multifaceted risks. 9,10 Among risk factors are technical, e.g., accuracy, reliability (data quality) and data security; 11–13 ethical, e.g., privacy, equity, accessibility and informed consent, but also fairness and bias; 14–16 and organizational, e.g., workforce displacement, acceptance, liability, and trust. Currently, a large proportion of AI systems are targeting decision support, running the risk of overreliance and automation-bias, as well as ignorance due to information or alarm fatigue. 17 In addition to erroneous algorithmic results, these scenarios could potentially lead to unsound medical practice or liability -Abstract Truncated-
health care sciences & services,health policy & services