How to bridge innovation and regulation for responsible AI in healthcare

Brian Anderson

DOI: https://doi.org/10.1038/s41591-024-02983-y

IF: 82.9

2024-05-18

Nature Medicine

Abstract:Nature Medicine, Published online: 17 May 2024; doi:10.1038/s41591-024-02983-y Quality assurance labs should independently test AI health tools.

biochemistry & molecular biology,cell biology,medicine, research & experimental

What problem does this paper attempt to address?

To do no harm — and the most good — with AI in health care

Carey Beth Goldberg,Laura Adams,David Blumenthal,Patricia Flatley Brennan,Noah Brown,Atul J. Butte,Morgan Cheatham,Dave deBronkart,Jennifer Dixon,Jeffrey Drazen,Barbara J. Evans,Sara M. Hoffman,Chris Holmes,Peter Lee,Arjun Kumar Manrai,Gilbert S. Omenn,Jonathan B. Perlin,Rachel Ramoni,Guillermo Sapiro,Rupa Sarkar,Harpreet Sood,Effy Vayena,Isaac S. Kohane,Emily Alsentzer,Brian Anderson,Ran D. Balicer,Andrew L. Beam,Erwin Bottinger,Rebecca W. Brendel,Payal Chandak,Arnaub Chatterjee,Michael Chernew,Antoine Duclos,Lee A. Fleisher,William Gordon,Elizabeth Healey,Alejandro Hernández-Arango,Michele Kathleen Herndon,Alexander Hoffmann,Edward M. Hundert,Indra Joshi,Tobias Kowatsch,Stephen Kraus,Lisa Soleymani Lehmann,Karim Lekadir,Vincent X. Liu,Daniel J. Nigrin,Reut Ohana,Nigam H. Shah,Haris Shuaib,Tania Simoncelli,Amelia Li Min Tan,Donrich Thaldar,Eugene Tunik,Tommy Wang,John Wilbanks,Yuchen Xu,Jianfei Zhao,the RAISE Consortium

DOI: https://doi.org/10.1038/s41591-024-02853-7

IF: 82.9

2024-02-24

Nature Medicine

Abstract:Drawing from real-life scenarios and insights shared at the RAISE (Responsible AI for Social and Ethical Healthcare) conference, we highlight the critical need for AI in health care (AIH) to primarily benefit patients and address current shortcomings in health care systems such as medical errors and access disparities.

biochemistry & molecular biology,cell biology,medicine, research & experimental
Regulatory Aspects of Artificial Intelligence and Machine Learning

Liron Pantanowitz,Matthew Hanna,Joshua Pantanowitz,Joe Lennerz,Walter H Henricks,Peter Shen,Bruce Quinn,Shannon Bennet,Hooman H Rashidi

DOI: https://doi.org/10.1016/j.modpat.2024.100609

2024-09-12

Abstract:In the realm of health care, numerous generative and nongenerative artificial intelligence and machine learning (AI-ML) tools have been developed and deployed. Simultaneously, manufacturers of medical devices are leveraging AI-ML. However, the adoption of AI in health care raises several concerns, including safety, security, ethical biases, accountability, trust, economic impact, and environmental effects. Effective regulation can mitigate some of these risks, promote fairness, establish standards, and advocate for more sustainable AI practices. Regulating AI tools not only ensures their safe and effective adoption but also fosters public trust. It is important that regulations remain flexible to accommodate rapid advances in this field to support innovation and also not to add additional burden to some of our preexisting and well-established frameworks. This study covers regional and global regulatory aspects of AI-ML including data privacy, software as a medical device, agency approval and clearance pathways, reimbursement, and laboratory-developed tests.
Regulating Artificial Intelligence and Machine Learning-Enabled Medical Devices in Europe and the United Kingdom

Phoebe Li,Robin Williams,Stephen Gilbert,Stuart Anderson

DOI: https://doi.org/10.5204/lthj.3073

2023-11-21

Abstract:Recent achievements in respect of Artificial Intelligence (AI) open up opportunities for new tools to assist medical diagnosis and care delivery. However, the typical process for the development of AI is through repeated cycles of learning and implementation, something that poses challenges to our existing system of regulating medical devices. Product developers face tensions between the benefits of continuous improvement/deployment of algorithms and keeping products unchanged. The latter more easily facilitates collecting evidence for safety assurance processes but sacrifices optimisation of performance and adaptation to user needs gained through learning-implementation cycles. The challenge is how to balance potential benefits with the need to assure their safety. Governance and assurance processes are needed that can accommodate real-time or near-real-time machine learning. Such an approach is of great importance in healthcare and other fields of application. AI has stimulated an intense process of learning as this new technology embeds in application contexts. The process is not only about the application of AI in the real world but also about the institutional arrangements for its safe and dependable deployment, including regulatory experimentation involving new market pathways, monitoring and surveillance, and sandbox schemes. We review the key themes, challenges and potential solutions raised at two stakeholder workshops and highlight recent attempts to adapt the laws for AI-enabled medical devices (AIeMD) with a special focus on the regulatory proposals in the UK and internationally. The UK regulatory trajectory shows signs of alignment with the US thinking, and yet the European Union model is still the most closely aligned framework.
Trustworthy Augmented Intelligence in Health Care

Elliott Crigger,Karen Reinbold,Chelsea Hanson,Audiey Kao,Kathleen Blake,Mira Irons

DOI: https://doi.org/10.1007/s10916-021-01790-z

IF: 4.92

2022-01-12

Journal of Medical Systems

Abstract:Abstract Augmented Intelligence (AI) systems have the power to transform health care and bring us closer to the quadruple aim: enhancing patient experience, improving population health, reducing costs, and improving the work life of health care providers. Earning physicians' trust is critical for accelerating adoption of AI into patient care. As technology evolves, the medical community will need to develop standards for these innovative technologies and re-visit current regulatory systems that physicians and patients rely on to ensure that health care AI is responsible, evidence-based, free from bias, and designed and deployed to promote equity. To develop actionable guidance for trustworthy AI in health care, the AMA reviewed literature on the challenges health care AI poses and reflected on existing guidance as a starting point for addressing those challenges (including models for regulating the introduction of innovative technologies into clinical care).

health care sciences & services,medical informatics
Leveraging law and ethics to promote safe and reliable AI/ML in healthcare

Katherine Drabiak

DOI: https://doi.org/10.3389/fnume.2022.983340

2022-09-27

Abstract:Artificial intelligence and machine learning (AI/ML) is poised to disrupt the structure and delivery of healthcare, promising to optimize care clinical care delivery and information management. AI/ML offers potential benefits in healthcare, such as creating novel clinical decision support tools, pattern recognition software, and predictive modeling systems. This raises questions about how AI/ML will impact the physician-patient relationship and the practice of medicine. Effective utilization and reliance on AI/ML also requires that these technologies are safe and reliable. Potential errors could not only pose serious risks to patient safety, but also expose physicians, hospitals, and AI/ML manufacturers to liability. This review describes how the law provides a mechanism to promote safety and reliability of AI/ML systems. On the front end, the Food and Drug Administration (FDA) intends to regulate many AI/ML as medical devices, which corresponds to a set of regulatory requirements prior to product marketing and use. Post-development, a variety of mechanisms in the law provide guardrails for careful deployment into clinical practice that can also incentivize product improvement. This review provides an overview of potential areas of liability arising from AI/ML including malpractice, informed consent, corporate liability, and products liability. Finally, this review summarizes strategies to minimize risk and promote safe and reliable AI/ML.
Regulating the Safety of Health-Related Artificial Intelligence

Anna Goldenberg,Devin Singh,Michael Silva,Colleen Flood

DOI: https://doi.org/10.12927/hcpol.2022.26824

2022-05-23

Healthcare Policy | Politiques de Santé

Abstract:This article analyzes whether Canada's present approach to regulating health-related artificial intelligence (AI) can address relevant safety-related challenges. Focusing primarily on Health Canada's regulation of medical devices with AI, it examines whether the existing regulatory approach can adequately address general safety concerns, as well as those related to algorithmic bias and challenges posed by the intersections of these concerns with privacy and security interests. It identifies several issues and proposes reforms that aim to ensure that Canadians can access beneficial AI while keeping unsafe products off Canadian markets and motivating safe, effective use of AI products for appropriate purposes and populations.
Harmonizing Artificial Intelligence Governance; A Model for Regulating a High-risk Categories and Applications in Clinical Pathology: The Evidence and some Concerns

Omabe Maxwell

DOI: https://doi.org/10.29328/journal.apcr.1001040

2024-03-18

Archives of Pathology and Clinical Research

Abstract:The Canadian healthcare system, grappling with issues like systemic and intelligently established structural anti-black racism, including indigenous nations; even within Pathology and Laboratory Medicine Communities: and deteriorating outcomes, sees potential in AI to address challenges, though concerns exist regarding exacerbating discriminatory practices. In clinical pathology, AI demonstrates superior diagnostic accuracy compared to pathologists in a study, emphasizing its potential to improve healthcare. However, AI governance is crucial to navigating ethical, legal, and societal concerns. The Royal College of Physicians of Canada acknowledges the transformative impact of AI in healthcare but stresses the need for responsible AI tools co-developed by diverse teams. Despite positive attitudes towards AI in healthcare, concerns about patient safety, privacy, and autonomy highlight the necessity for comprehensive education, engagement, and collaboration. Legal concerns, including liability and regulation, pose challenges, emphasizing the need for a robust regulatory framework. AI application in healthcare is categorized as high-risk, demanding stringent regulation to ensure safety, efficacy, and fairness. A parallel is drawn to drug regulation processes, suggesting a similar approach for AI. The lack of transparency in AI-based decision-making raises ethical questions, necessitating measures to address biases and ensure patient privacy. Social accountability is crucial to prevent AI from exacerbating health disparities and harming marginalized communities. In conclusion, while AI offers potential benefits in clinical pathology, a cautious approach with comprehensive governance measures is essential to mitigate risks and ensure ethical AI integration into healthcare.
Innovation and challenges of artificial intelligence technology in personalized healthcare

Yu-Hao Li,Yu-Lin Li,Mu-Yang Wei,Guang-Yu Li

DOI: https://doi.org/10.1038/s41598-024-70073-7

IF: 4.6

2024-08-19

Scientific Reports

Abstract:As the burgeoning field of Artificial Intelligence (AI) continues to permeate the fabric of healthcare, particularly in the realms of patient surveillance and telemedicine, a transformative era beckons. This manuscript endeavors to unravel the intricacies of recent AI advancements and their profound implications for reconceptualizing the delivery of medical care. Through the introduction of innovative instruments such as virtual assistant chatbots, wearable monitoring devices, predictive analytic models, personalized treatment regimens, and automated appointment systems, AI is not only amplifying the quality of care but also empowering patients and fostering a more interactive dynamic between the patient and the healthcare provider. Yet, this progressive infiltration of AI into the healthcare sphere grapples with a plethora of challenges hitherto unseen. The exigent issues of data security and privacy, the specter of algorithmic bias, the requisite adaptability of regulatory frameworks, and the matter of patient acceptance and trust in AI solutions demand immediate and thoughtful resolution .The importance of establishing stringent and far-reaching policies, ensuring technological impartiality, and cultivating patient confidence is paramount to ensure that AI-driven enhancements in healthcare service provision remain both ethically sound and efficient. In conclusion, we advocate for an expansion of research efforts aimed at navigating the ethical complexities inherent to a technology-evolving landscape, catalyzing policy innovation, and devising AI applications that are not only clinically effective but also earn the trust of the patient populace. By melding expertise across disciplines, we stand at the threshold of an era wherein AI's role in healthcare is both ethically unimpeachable and conducive to elevating the global health quotient.

multidisciplinary sciences
Gaps in the Global Regulatory Frameworks for the Use of Artificial Intelligence (AI) in the Healthcare Services Sector and Key Recommendations

Kavitha Palaniappan,Elaine Yan Ting Lin,Silke Vogel,John C. W. Lim

DOI: https://doi.org/10.3390/healthcare12171730

2024-09-03

Health Care

Abstract:Artificial Intelligence (AI) has shown remarkable potential to revolutionise healthcare by enhancing diagnostics, improving treatment outcomes, and streamlining administrative processes. In the global regulatory landscape, several countries are working on regulating AI in healthcare. There are five key regulatory issues that need to be addressed: (i) data security and protection—measures to cover the "digital health footprints" left unknowingly by patients when they access AI in health services; (ii) data quality—availability of safe and secure data and more open database sources for AI, algorithms, and datasets to ensure equity and prevent demographic bias; (iii) validation of algorithms—mapping of the explainability and causability of the AI system; (iv) accountability—whether this lies with the healthcare professional, healthcare organisation, or the personified AI algorithm; (v) ethics and equitable access—whether fundamental rights of people are met in an ethical manner. Policymakers may need to consider the entire life cycle of AI in healthcare services and the databases that were used for the training of the AI system, along with requirements for their risk assessments to be publicly accessible for effective regulatory oversight. AI services that enhance their functionality over time need to undergo repeated algorithmic impact assessment and must also demonstrate real-time performance. Harmonising regulatory frameworks at the international level would help to resolve cross-border issues of AI in healthcare services.

health care sciences & services,health policy & services
FDA Perspective on the Regulation of Artificial Intelligence in Health Care and Biomedicine.

Haider J. Warraich,R. Califf,Troy Tazbaz

DOI: https://doi.org/10.1001/jama.2024.21451

2024-10-15

Abstract:Importance Advances in artificial intelligence (AI) must be matched by efforts to better understand and evaluate how AI performs across health care and biomedicine as well as develop appropriate regulatory frameworks. This Special Communication reviews the history of the US Food and Drug Administration's (FDA) regulation of AI; presents potential uses of AI in medical product development, clinical research, and clinical care; and presents concepts that merit consideration as the regulatory system adapts to AI's unique challenges. Observations The FDA has authorized almost 1000 AI-enabled medical devices and has received hundreds of regulatory submissions for drugs that used AI in their discovery and development. Health AI regulation needs to be coordinated across all regulated industries, the US government, and with international organizations. Regulators will need to advance flexible mechanisms to keep up with the pace of change in AI across biomedicine and health care. Sponsors need to be transparent about and regulators need proficiency in evaluating the use of AI in premarket development. A life cycle management approach incorporating recurrent local postmarket performance monitoring should be central to health AI development. Special mechanisms to evaluate large language models and their uses are needed. Approaches are necessary to balance the needs of the entire spectrum of health ecosystem interests, from large firms to start-ups. The evaluation and regulatory system will need to focus on patient health outcomes to balance the use of AI for financial optimization for developers, payers, and health systems. Conclusions and Relevance Strong oversight by the FDA protects the long-term success of industries by focusing on evaluation to advance regulated technologies that improve health. The FDA will continue to play a central role in ensuring safe, effective, and trustworthy AI tools to improve the lives of patients and clinicians alike. However, all involved entities will need to attend to AI with the rigor this transformative technology merits.

Medicine,Law,Computer Science
How AI challenges the medical device regulation: patient safety, benefits, and intended uses

Daria Onitiu,Sandra Wachter,Brent Mittelstadt

DOI: https://doi.org/10.1093/jlb/lsae007

IF: 3.4

2024-04-09

Journal of Law and the Biosciences

Abstract:Abstract This article examines whether the EU Medical Device Regulation (MDR) adequately addresses the novel risks of AI-based medical devices (AIaMDs), focusing on AI medical imaging tools. It examines two questions: first, does the MDR effectively deal with issues of adaptability, autonomy, bias, opacity, and the need of trustworthiness of AIaMD? Second, does the manufacturer’s translation of the MDR’s requirements close a discrepancy between an AIaMDs’ expected benefit and the actual clinical utility of assessing device safety and effectiveness beyond the narrow performance of algorithms? While the first question has previously received attention in scholarly literature on regulatory and policy tensions on AIaMD generally, and work on future technical standard setting, the second has been comparatively overlooked. We argue that effective regulation of AIaMD requires framing notions of patient safety and benefit within the manufacturer’s articulation of the device’s intended use, as well as reconciling tensions. These tensions are on (i) patient safety and knowledge gaps surrounding fairness, (ii) trustworthiness and device effectiveness, (iii) the assessment of clinical performance, and (iv) performance updates. Future guidance needs to focus on the importance of translated benefits, including nuanced risk framing and looking at how the limitations of AIaMD inform the intended purpose statement in the MDR.

ethics,law,medicine, legal,medical ethics
Learning From Experience and Finding the Right Balance in the Governance of Artificial Intelligence and Digital Health Technologies

Stephen Gilbert,Stuart Anderson,Martin Daumer,Phoebe Li,Tom Melvin,Robin Williams

DOI: https://doi.org/10.2196/43682

IF: 7.076

2023-04-14

Journal of Medical Internet Research

Abstract:Artificial intelligence (AI) and machine learning medical tools have the potential to be transformative in care delivery; however, this change will only be realized if accompanied by effective governance that ensures patient safety and public trust. Recent digital health initiatives have called for tighter governance of digital health. A correct balance must be found between ensuring product safety and performance while also enabling the innovation needed to deliver better approaches for patients and affordable efficient health care for society. This requires innovative, fit-for-purpose approaches to regulation. Digital health technologies, particularly AI-based tools, pose specific challenges to the development and implementation of functional regulation. The approaches of regulatory science and “better regulation” have a critical role in developing and evaluating solutions to these problems and ensuring effective implementation. We describe the divergent approaches of the European Union and the United States in the implementation of new regulatory approaches in digital health, and we consider the United Kingdom as a third example, which is in a unique position of developing a new post-Brexit regulatory framework.

health care sciences & services,medical informatics
How to support the transition to AI-powered healthcare

DOI: https://doi.org/10.1038/s41591-024-02897-9

IF: 82.9

2024-03-20

Nature Medicine

Abstract:To make health systems more sustainable in the long-term, incentivize artificial intelligence (AI) and digital technologies that are grounded on careful testing and real-world validation.

biochemistry & molecular biology,cell biology,medicine, research & experimental
Responsibility and Regulation: Exploring Social Measures of Trust in Medical AI

Glenn McGarry,Andy Crabtree,Lachlan Urquhart,Alan Chamberlain

DOI: https://doi.org/10.1145/3686038.3686041

2024-08-05

Abstract:This paper explores expert accounts of autonomous systems (AS) development in the medical device domain (MD) involving applications of artificial intelligence (AI), machine learning (ML), and other algorithmic and mathematical modelling techniques. We frame our observations with respect to notions of responsible innovation (RI) and the emerging problem of how to do RI in practice. In contribution to the ongoing discourse surrounding trustworthy autonomous system (TAS) [29], we illuminate practical challenges inherent in deploying novel AS within existing governance structures, including domain specific regulations and policies, and rigorous testing and development processes, and discuss the implications of these for the distribution of responsibility in novel AI deployment.

Human-Computer Interaction
Doctor, Bot, or Both: Questioning the Medicolegal Liability of Artificial Intelligence in Indian Healthcare

Satvik N Pai,Madhan Jeyaraman,Naveen Jeyaraman,Sankalp Yadav

DOI: https://doi.org/10.7759/cureus.69230

2024-09-11

Cureus

Abstract:Artificial intelligence (AI) is rapidly transforming the landscape of healthcare, with its applications ranging from diagnostics to robotic surgery. As India swiftly integrates AI into its healthcare system, a critical question arises: who bears the liability when AI-driven medical decisions go wrong? This article explores the complexities of assigning legal responsibility for AI in healthcare, particularly within the Indian context. While India's legal framework, including the Information Technology Act, intersects with AI usage, it lacks specific provisions addressing AI liability. The National Strategy for AI, although a step forward, does not resolve the ambiguities surrounding accountability. The article also examines how other countries, particularly the European Union and the United States, are beginning to tackle these challenges through proposed directives and regulations. However, the absence of clear, comprehensive laws worldwide reflects the inherent difficulties in legislating AI stemming from its autonomous nature, the "black box" problem, and the multifaceted roles of stakeholders. Until robust legal frameworks are established, the article cautions healthcare professionals to exercise vigilance when using AI tools, recognizing that they may still bear significant liability. This evolving legal landscape underscores the need for continued deliberations in AI regulations.
Trust and Medical AI: The challenges we face and the expertise needed to overcome them

Thomas P. Quinn,Manisha Senadeera,Stephan Jacobs,Simon Coghlan,Vuong Le

DOI: https://doi.org/10.48550/arXiv.2008.07734

2020-08-18

Abstract:Artificial intelligence (AI) is increasingly of tremendous interest in the medical field. However, failures of medical AI could have serious consequences for both clinical outcomes and the patient experience. These consequences could erode public trust in AI, which could in turn undermine trust in our healthcare institutions. This article makes two contributions. First, it describes the major conceptual, technical, and humanistic challenges in medical AI. Second, it proposes a solution that hinges on the education and accreditation of new expert groups who specialize in the development, verification, and operation of medical AI technologies. These groups will be required to maintain trust in our healthcare institutions.

Artificial Intelligence,Computers and Society
Liability for harm caused by AI in healthcare: an overview of the core legal concepts

Dane Bottomley,Donrich Thaldar

DOI: https://doi.org/10.3389/fphar.2023.1297353

2023-12-14

Abstract:The integration of artificial intelligence (AI) into healthcare in Africa presents transformative opportunities but also raises profound legal challenges, especially concerning liability. As AI becomes more autonomous, determining who or what is responsible when things go wrong becomes ambiguous. This article aims to review the legal concepts relevant to the issue of liability for harm caused by AI in healthcare. While some suggest attributing legal personhood to AI as a potential solution, the feasibility of this remains controversial. The principal-agent relationship, where the physician is held responsible for AI decisions, risks reducing the adoption of AI tools due to potential liabilities. Similarly, using product law to establish liability is problematic because of the dynamic learning nature of AI, which deviates from static products. This fluidity complicates traditional definitions of product defects and, by extension, where responsibility lies. Exploring alternatives, risk-based determinations of liability, which focus on potential hazards rather than on specific fault assignments, emerges as a potential pathway. However, these, too, present challenges in assigning accountability. Strict liability has been proposed as another avenue. It can simplify the compensation process for victims by focusing on the harm rather than on the fault. Yet, concerns arise over the economic impact on stakeholders, the potential for unjust reputational damage, and the feasibility of a global application. Instead of approaches based on liability, reconciliation holds much promise to facilitate regulatory sandboxes. In conclusion, while the integration of AI systems into healthcare holds vast potential, it necessitates a re-evaluation of our legal frameworks. The central challenge is how to adapt traditional concepts of liability to the novel and unpredictable nature of AI-or to move away from liability towards reconciliation. Future discussions and research must navigate these complex waters and seek solutions that ensure both progress and protection.
Challenges and Opportunities for AI in Healthcare

Kan Yekaterina

DOI: https://doi.org/10.59022/ijlp.203

2024-07-30

International Journal of Law and Policy

Abstract:The integration of artificial intelligence (AI) in healthcare presents a dual challenge: maximizing the efficiency of medical processes while safeguarding patient privacy. This comprehensive review examines the delicate balance between leveraging AI's potential in healthcare and preserving individual data privacy. Through analysis of recent literature, case studies, and regulatory frameworks, we explore the current landscape of AI applications in healthcare, associated privacy risks, and emerging solutions. Findings reveal that while AI significantly enhances diagnostic accuracy and treatment planning, it also raises concerns about data security and patient confidentiality. Key challenges include ensuring GDPR and HIPAA compliance, managing large-scale health data, and maintaining transparency in AI decision-making processes. Promising approaches such as federated learning and differential privacy emerge as potential solutions. This review underscores the need for a multidisciplinary approach involving healthcare providers, AI developers, ethicists, and policymakers to create robust, privacy-preserving AI systems in healthcare.
Racing into the fourth industrial revolution: exploring the ethical dimensions of medical AI and rights-based regulatory framework

Simisola Johnson

DOI: https://doi.org/10.1007/s43681-022-00153-9

2022-02-01

AI and Ethics

Abstract:Artificial intelligence (AI) is playing an increasing role in clinical decision-making. This capability is learned and therefore, the ways in which data are coded, collected, and selected to train algorithms, including the tools overall design, impact patterns that AI notices and its output. Ethically, racial health inequalities are unjust—this is the priority position of Rawls ethical theory of justice-and erodes trust in the medical system. To close gaps, operationalising fairness and equity into AI-based clinical support tools and decision-making algorithms entails seeking to maximize the benefits of AI while preventing harm and minimizing intervention-generated inequalities among historically underserved and underrepresented populations in medicine. Equitable healthcare is an idea whose time has come—as is the time for swift action on AI regulations to protect rights—and people. Alas, the triple aim of this commentary is to cover ethical implications of algorithmic bias, practical boundaries involving theoretical frameworks to guide the design and development of equitable, decision-making AI and why a rights-based approach to AI regulation enhances health for all.
Distributed Governance of Medical AI

William Nicholson Price II

DOI: https://doi.org/10.2139/ssrn.4051834

2022-01-01

SSRN Electronic Journal

Abstract:Artificial intelligence (AI) promises to bring substantial benefits to medicine. In addition to pushing the frontiers of what is humanly possible, like predicting kidney failure or sepsis before any human can notice, it can democratize expertise beyond the circle of highly specialized practitioners, like letting generalists diagnose diabetic degeneration of the retina. But AI doesn’t always work, and it doesn’t always work for everyone, and it doesn’t always work in every context. AI is likely to behave differently in well-resourced hospitals where it is developed than in poorly resourced frontline health environments where it might well make the biggest difference for patient care. To make the situation even more complicated, AI is unlikely to go through the centralized review and validation process that other medical technologies undergo, like drugs and most medical devices. Even if it did go through those centralized processes, ensuring high-quality performance across a wide variety of settings, including poorly resourced settings, is especially challenging for such centralized mechanisms. What are policymakers to do? This short Essay argues that the diffusion of medical AI, with its many potential benefits, will require policy support for a process of distributed governance, where quality evaluation and oversight take place in the settings of application—but with policy assistance in developing capacities and making that oversight more straightforward to undertake. Getting governance right will not be easy (it never is), but ignoring the issue is likely to leave benefits on the table and patients at risk.

How to bridge innovation and regulation for responsible AI in healthcare

To do no harm — and the most good — with AI in health care

Regulatory Aspects of Artificial Intelligence and Machine Learning

Regulating Artificial Intelligence and Machine Learning-Enabled Medical Devices in Europe and the United Kingdom

Trustworthy Augmented Intelligence in Health Care

Leveraging law and ethics to promote safe and reliable AI/ML in healthcare

Regulating the Safety of Health-Related Artificial Intelligence

Harmonizing Artificial Intelligence Governance; A Model for Regulating a High-risk Categories and Applications in Clinical Pathology: The Evidence and some Concerns

Innovation and challenges of artificial intelligence technology in personalized healthcare

Gaps in the Global Regulatory Frameworks for the Use of Artificial Intelligence (AI) in the Healthcare Services Sector and Key Recommendations

FDA Perspective on the Regulation of Artificial Intelligence in Health Care and Biomedicine.

How AI challenges the medical device regulation: patient safety, benefits, and intended uses

Learning From Experience and Finding the Right Balance in the Governance of Artificial Intelligence and Digital Health Technologies

How to support the transition to AI-powered healthcare

Responsibility and Regulation: Exploring Social Measures of Trust in Medical AI

Doctor, Bot, or Both: Questioning the Medicolegal Liability of Artificial Intelligence in Indian Healthcare

Trust and Medical AI: The challenges we face and the expertise needed to overcome them

Liability for harm caused by AI in healthcare: an overview of the core legal concepts

Challenges and Opportunities for AI in Healthcare

Racing into the fourth industrial revolution: exploring the ethical dimensions of medical AI and rights-based regulatory framework

Distributed Governance of Medical AI