Guardrails for avoiding harmful medical product recommendations and off-label promotion in generative AI models

Daniel Lopez-Martinez
2024-06-24
Abstract:Generative AI (GenAI) models have demonstrated remarkable capabilities in a wide variety of medical tasks. However, as these models are trained using generalist datasets with very limited human oversight, they can learn uses of medical products that have not been adequately evaluated for safety and efficacy, nor approved by regulatory agencies. Given the scale at which GenAI may reach users, unvetted recommendations pose a public health risk. In this work, we propose an approach to identify potentially harmful product recommendations, and demonstrate it using a recent multimodal large language model.
Artificial Intelligence
What problem does this paper attempt to address?
The paper primarily focuses on a potential issue that may arise from the application of generative artificial intelligence (GenAI) models in the medical field: these models may recommend unapproved or unverified drug uses (referred to as "off-label" use). Since GenAI models are typically trained on general datasets containing a large amount of diverse data and may lack sufficient human supervision during development, they have the potential to learn and recommend drug uses that have not been fully evaluated for safety and efficacy. The paper proposes a method to identify potentially harmful drug recommendations in the outputs of GenAI models and demonstrates this through a recent multimodal large language model. Specifically, the researchers designed a process that includes steps such as input standardization, named entity recognition, product and indication identification, and off-label use identification. In this way, it is possible to detect whether there is off-label promotion in the model's responses and further assess its potential harm. Overall, this paper aims to highlight an important ethical and legal issue of GenAI in medical applications and proposes a preliminary method to address this issue, thereby reducing the potential risk to public health.